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SUMMARY 


This report presents the results of a one-year research and development 
program in the area of decision training. The project covers two areas of 
the Navy SF-2F Light Airborne Multi-Purpose System (LAMPS) Anti-Submarine 
Warfare (ASW) operation: emergency and tactical operations. This report 
describes the unclassified results of the project. Other materials have 
been presented in a separate, classified report. 


This report is organized into five chapters. The first chapter is an 
introduction which includes an overview of the project, background, and 
' technical approach. Chapters 2, 3, and 4 correspond to the first three 
phases of the program. The results of the job analysis phase is in Chapter 
2. In this phase, the tasks involved in the Navy SH-2F (LAMPS) helicopter 
training system were analyzed, decision tasks were identified, and a number 
of the identified tasks were selected for further analysis. Chapter 3 
describes the second phase of the project: decision task analysis. The 
selection decision tasks were analyzed to identify the required facts, the 
underlying concepts, the decision rules, and the step-by-step procedures 
that determine the optimal choice of alternatives. Based on the results 
of this analysis, a system for instructional guideline generation was 
developed. The instructional guideline generation system is presented in 
Chapter 4. The system provides a procedure to generate essential information 
to be incorporated in the production of instructional materials for any 
decision task subject to training in any specific instructional environment. 
Each of the chapters 2 through 4 starts with a presentation of the general 
approach and activities associated with the corresponding phase. The 
following sections cover the methodology designed and utilized in the 
corresponding phase of the project. Technical details and the results of 
the phase appears in the remaining sections. The last chapter of the report 
presents the conclusions of the project. The dimensions for improvement of 
the system developed during the project are identified and a computerized 
instructional guideline generated aid, as a means to achieve improvements 
in all of these dimensions, is briefly described. Also included are 
suggestions for decision aids based on the results of the analyses, an 
evaluation plan for the instructional materials developed for decision tasks, 
and the results of the instructional guideline generation system. The primary 
results of the analysis and decision trees appear in the appendices. 


During the one year project period, a system to generate instructional 
guidelines for a task-specific decision training program, in any training 
environment, was developed. The system selects the most effective 
instructional method and media and provides required instructional contents 
for any specific decision task, subject to training in a specific training 
environment. The project has been aimed not only to develop a systematic 
procedure for providing instructional guidelines for decision tasks 
encountered in LAMPS ASW operation, but also to design a task-independent 
methodology in order to make application of the system in other task 
domains subject to minimal modification effort. Such a methodology allows 
incorporation of general purpose decision training components into task 
specific decision training programs. This incorporation provides trainees 
with the basic skills required for making task-specific decisions within 

a new domain, in case of their transfer to other task domains. Furthermore, 
Since the characteristics of decision tasks within any specific domain are 
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subject to change with time, the methodology will allow development of more 
effective training programs even in the case where there is no possibility 
of transferring the trainees to other task domains. Finally, the methodology 
will provide for carry over of the decision making capability into incidential 


functions, which are usually left untrained. 
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PREFACE 


Decision task training is currently not receiving required emphasis 
in formal training environments, most probably due to a lack of appropriate 
training technology. The training of decision tasks still takes place on 
the job, a costly process in terms of dollars and time. 


The work summarized in this technical report represents an attempt 
to meld decision theoretic concepts with operational knowledge in order 
to advance the state-of-the-art in the area of decision task training. 
It is anticipated that the procedures which have been developed under 
this contract, after subsequent validation and refinement, will be 
incorporated into the Instructional Systems Development (ISD) process. 


Appreciation is expressed to the following individuals who 
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SECTION I 
INTRODUCTION | 
OVERVIEW 


A one-year research and development program in the area of decision 
training is described in this report. The program investigated the use of 
decision analysis techniques in the design of instructional materials for 
decision training in military systems. The long range goal is to define a 
decision training approach that can be applied to a wide variety of system 
training courses. As a first step toward that goal, the present program 
focused on the decision training requirements of the Navy SH-2F Light Airborne 
Multi-Purpose System (LAMPS) Anti-Submarine Warfare (ASW) helicopter. In this 
environment, decision analysis techniques were used to (1) identify and 
select critical decision making situations during helicopter operation, (2) 
define the decision making tasks required of the Pilot, Airborne Tactical 
Officer (ATO), and the Sensor Operator, and (3) specify the skills and 
knowledge necessary to make effective decisions. 


To an ever-increasing extent, the operators of advanced military systems, 
such as aircraft, ships, weapons, C3, etc., act primarily as decision makers. 
That is, the operator's most important functions are to select from alternative 
data sources and to choose among alternative courses of action. Although he 
may be well trained to carry out each alternative procedure, he is virtually 
never trained in how to most effectively make the crucial decision among them. 
Traditionally, acquiring the judgment to make sound decisions has been 
considered an art, which must be developed over years of experience, or 
through long apprenticeship to a veteran decision maker. This approach, 
aside from being of dubious validity, is inappropriate in today's military 
environment, where an operator must step into a job and function well from 
the start. 


Within the past several years, the general rules of effective decision 
making have been consolidated into the technical area termed "decision 
analysis." During the same period, the systems approach to training has 
established a generalized methodology for dealing with diverse content areas 
in the design of training courses called Instructional System Development 
(ISD). Accordingly, it is the objective here to bring these two areas 
together in order to define the decision training objectives of typical 
military systems and to specify ways of meeting them. Successful joining 
of the two technologies should have a profound effect on future system 
performance and lifetime cost. 


The current program has the following major objectives: 


a. To analyze a specific military training system, the SH-2F (LAMPS) 
helicopter, in terms of specific decision analysis techniques. 


b. To identify the decision situations that occur during regular and 
emergency system operations. 
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c. To isolate the decision tasks that are required to resolve a 
selected set of decision situations. 


d. To identify, for each decision task, (a) the required facts, 
(b) the underlying concepts, (c) the decision rules, and (d) the 
step-by-step procedures that determine the optimal choice of 
alternatives. 


e. To contribute to the production of training materials for one 
or more of the decision tasks, to include: (a) training analysis, 
and (b) production of instruction. 


f. To evaluate the materials produced for this test case, and to 
analyze the implications for (a) similar system-oriented courseware, 
(b) courseware for generalized decision training, and (c) decision- 
aiding software for the SH-2F and related systems. 


RATIONALE 


The term “decision making" has been used to describe a wide range of 
cognitive processes. Building upon the classification of others, Nickerson 
and Feehrer (1975) have conceptualized decision making as a collection of 
problem solving or decision tasks which involve: “information gathering, 
data evaluation, hypothesis generation, problem structuring, hypothesis 
evaluation, preference specification, action selection, and decision evalua- 
tion." 


Study of human decision making behavior reveals deficiencies accom- 
plished with the different decision making components (Hammell and Mara, 
1970). In the literature, the term "deficiencies" has been used in two ways: 
(1) to refer to stereotypes ways of behaving suboptimally, such as the 
tendency of humans to be overly conservative in their application of proba- 
bilitistic information to the evaluation of hypotheses, and (2) to refer to 
basic human cognitive limitations such as memory, attention span, and 
processing limits such as the inability of most people to weigh more than 
some small number of factors in arriving at a preference among alternatives 
without procedural help. 


Different approaches to correct deficiencies in human decision making 
process have been suggested (Schrenk, 1969). There are basically three ways 
to improve the performance of the human decision element in a system: 


a. Selection: insure that decisions are made only by individuals 
who are competent to make them. 


b. Decision Aiding: provide decision makers with procedural and 
technical aids to compensate for their own limitations. 


c. Decision Training: attempt to improve the decision-related 
skills of people in decision-making positions. 


Decision aiding and decision training can be viewed as complementary to one 
another. While decision training attempts to improve decision making behavior 
by training out deficiencies and highlighting limitations, decision aiding 
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provides the decision maker with procedural and technical aids which let him 
go beyond his own limitations in the process of decision making. Much of the 
research work has been performed on decision aiding while decision training 
has been rarely investigated. One objective of the present research program 
was to narrow this gap. 


There are a number of decision training programs that are being pursued 
with some success. These programs are either limited to the abstract decision 
making subprocesses without an integration of the decision training with the 
area of the trainee's expertise, or they highly concentrate on the task- 
specific knowledge and fail to provide the trainees with a competent gener- 
alized decision making behavior. Training programs based on the first 
approach do not provide the required link between decision training and the 
application area. Programs with the second approach suffer from their short- 
comings in three areas: 


a. Lack of decision training relevant to a new domain, in case of the 
trainee's transfer to other task domains. 


b. Loss of decision training effectiveness over time, since the 
characteristics of decision tasks within any specific domain 
are subject to change with time. 


c. Lack of carry over of the decision making capability into 
incidental functions, which are usually left untrained. 


To avoid the shortcomings of the existing programs, a decision training 
methodology must be developed that allows incorporation of general purpose 
decision training components into task specific decision training programs. 
Since many problems encountered in the development of such a decision 
training methodology have not been previously addressed, and since the 
development process involves a cross-section of different disciplines, such 
a development provided a desirable area for research and development. 


The research and development process provided for incorporation of 
different areas of knowledge into the development of an universal decision 
training methodology. These areas of knowledge include decision analysis, 
psychology, training, and computer science. Different components of a 
decision process were analyzed and the required behaviors involved in these 
components were identified. The cognitive complexity of each identified 
behavior was recognized and the training behavioral objectives were defined. 
The best instructional method, media, and content for accomplishing the 
training objectives were identified. 


CURRENT DECISION TRAINING PROGRAMS. There are a number of decision training 
programs that are being pursued with some success. These programs are either 
task specific or treat only limited aspects of the decision making process. 
Some of the programs are used in operational contexts. 


Einhorn and Hogard (1975) developed an approach to teaching multi- 
attribute utility analysis to top-level executives and middle-level managers. 
While their system, termed "An Idiot's Guide to Decision Making" (Goodman 
et al, 1976), maintains a reasonable degree of independence with respect to 
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any specific domain, it covers only one method of alternative evaluation. 
Other subprocesses involved in the decision making process, as well as the 
interrelationships among the subprocesses in the procedural body of the 
decision making system, are ignored. 


Decisions and Designs, Inc. (DDI) has developed a Rapid Screening 
Decision Training System. The system involves an interactive computer pro- 
gram that simplifies a decision analysis by focusing on a limited number of 
alternatives and the major causes of uncertainty (Selvidge, 1976). The 
training aspect of the system consists of displaying the expected value 
associated with each alternative evaluation and does not cover other elements 
of decision making such as problem recognition, alternative development, and 
the optimal sequencing and effort allocation for these subprocesses... 


Hammond, et al (1975) present judgment as the key elements of a decision 
making process. They assume that if people are taught the theory behind 
judgment analysis and then trained in increasingly difficult applications of 
task situations, they will eventually be able to analyze any problem properly. 
Based on this theory, their training system focuses strongly on judgmental 
aspects and ignores the other elements of the decision making process. 


The Decision Analysis Group at Stanford Research Institute (SRI) con- 
ducts several different kinds of training programs for decision makers 
(Goodman, et al, 1976). Through these programs, trainees are expected to 
learn that the decision theoretic methodology exists, that uncertainties 
and utilities can be quantitatively estimated and that they can begin to 
structure and work through their own decision problems. While the programs 
are enriched by a reasonable degree of generality and completeness, they 
do not provide the required link between decision training and the specific 
application area. 


Michigan State University's Medical School approach to the training of 
physicians is based on a total curriculum design wherein decision analysis 
is integrated within specific content areas (Allal, 1973; Elstein, Shulman, 
and Sprafka, 1977). It assumes that the diagnostic phase of medicine con- 
sists of generating hypotheses about what the medical problem might be, 
distinguishing the relevant from the irrelevant features of the case, and 
then systematically gathering information to test and compare alternative 
hypotheses. The program is highly task-specific and it relies heavily on 
the case-study method. 


The Los Angeles Police Academy's "shoot/no shoot" training is an example 
of a task-specific decision training program. Although the program covers no 
formal training in either probability theory or decision theory, it includes 
extensive courses in the established important attributes that should be 
considered when deciding whether or not to shoot in a given situation. 
There is no relative weighting scheme for the attributes nor decision rules 
that translate the utilities of the attributes into a decision. There are, 
however, general guidelines that help the cadets make the decision. Due to 
time criticality, the cadets are taught to prune the decision tree before 
the actual decision situation arises. The training system is tailored for 
the specific task involved and lacks the generality and completeness desired 
in a decision training program. 
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The review of current decision training programs seems to reveal a 
lack of completeness, generality, and structural guidance in the system 
under development or currently in use. These three features are essential 
to a decision training program with reasonable generality. A research and 
development project for a decision training methodology embodying these 
three features was included in the objectives of this program. 


REQUIREMENTS FOR A GENERALIZED DECISION TRAINING PROGRAM. Current efforts 
in the development of decision training systems have usually evolved in 

the context of task-specific systems (e.g., medical, tactical, managerial, 
etc.). A training system for decision makers that is developed within the 
context of such a task-specific domain, without incorporation of general 
purpose decision training components, does not provide trainees with the 
skills required for making task-specific decisions within other domains. 
Furthermore, since the characteristics of decision tasks within any specific 
domain may be subject to change with time, such training systems will not 
even provide useful long-term guidelines within the trainees' own domain. 

In these systems, the decision maker learns a tailor-made set of conceptual 
components, procedures, techniques, heuristic rules, parameters, and values 
unique to that particular system at the time of training. The set of 
concepts that the potential decision maker learns is in reality a very 
specific sub-set of more general conceptual components, procedures, and 
techniques. Yet, he never becomes aware that, in fact, it is a sub-set 

of the more general set because it is outside the training boundary. Hence, 
if a decision maker is transferred to a new decision making domain, or the 
characteristics of the decision tasks within his decision domain change, he 
typically has two choices: he must either enter a new task-specific decision 
training program (and learn another unique conceptual sub-set again without 
relaxing its relationship to the first sub-set learned), or he must learn 

to invent his own conceptual set without the benefit of a formal training 
program. In the latter case, the decision maker is without a formal guide 
and often attempts to "cram" his new world into a conceptual sub-set learned 
for an entirely different decision task. 


Considering the fact that the characteristics of decision tasks within 
any specific domain are subject to change with time, even decision training 
systems designed for specific domains must include components of general 
purpose decision training. A comparative analysis of general purpose versus 
task specific decision training will identify the required components of 
domain independent decision training that must be included in a task specific 
decision training system. Such a comparative analysis was included in the 
proposed program. 


BACKGROUND 


PROBLEM STATEMENT. Instructional System Development (ISD) currently provides 
a systematic and scientific methodology for (1) identifying the tasks that 
operators must perform, (2) specifying the hierarchy of training objectives, 
and (3) developing the procedures and schedules for training the operators 

to perform the identified tasks. However, decision making tasks have 
typically not been among those identified and treated by the ISD methodology. 
Decision tasks are thus neglected in present training courses. In most 
cases, alternative selection has been considered as a well-structured 
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"cookbook" response, rather than as a decision consisting of specific 
sequences of tasks to be trained. In those cases where critical decision 
points have been recognized, performance is assumed to come "from experience" 
rather than by explicit instruction. Research is definitely required to 
develop a methodology for identifying and isolating decision tasks in current 
system training functions, as well as for development procedures for train- 
ing decision making. 


DECISION MAKING. Investigation of human decision making has focused on the 
process of choosing among alternatives under various conditions of risk and 
uncertainty. Research has identified sequences of tasks included in the 
literature on decision making, the list given by Nickerson and Feehrer (1975) 
provides a comprehensive set for purposes of discussion and comparison. It 
consists of (1) information gathering, (2) data evaluation, (3) problem 
structuring, (4) hypothesis generation, (5) hypothesis evaluation, (6) 
preference specification, (7) action selection, and (8) decision evaluation. 


Theoretical research on normative (ideal) decision making has also 
identified sources of human decision-making deficiencies. Such deficiencies 
include stereo-typed choice sequences, conservative probability application, 
incomplete data acquisition, etc. Several investigators (Hammell and Mara, 
1970; Kanarick, 1969; Pesch, Hammel, and Ewalt, 1974) have suggested that 
decision training programs can diagnose and give instruction regarding deci- 
sion deficiencies. 


DECISION ANALYSIS. Technqgiues of decision analysis have been developed that 
can be used to identify the decision points in a sequence of action per- 
formed by a system operator. These techniques also provide a systematic 
method for identifying the decision parameters specific to the situation at 
hand (i.e., alternative, rules, attributes). Thus, the tasks and associated 
inputs of decision making can be explicitly defined for training purposes. 


The multi-attribute utility approach to analyzing decision situations 
in one example of these techniques (Gardiner and Edwards, 1975; Gardiner, 
1977). Essentially, this approach first establishes dimensions of value -- 
"attributes" for the relevant set of decision outcomes. Each outcome to 
be evaluated is located on each dimension -- i.e., assigned an attribute 
utility -- by means of observation of judgment. Utility can be thought 
of as gain to the operator; gains can be positive; negative gains are 
costs. An overall utility for the outcomes is achieved by aggregating the 
multiple utilities, most often by calculating a weighted average over 
attributes. The expected utility of an outcome is its aggregated utility 
multiplied by its probability of occuring, and the usual decision rule is 
to select the outcome with maximum expected utility, although in some cases 
other rules may be preferred. 


DECISION AIDING. Introduction of decision aids to the decision environment 
provides a framework for familiarizing the operator with decision theoreti- 
cal concepts and decision training approaches. It has been shown that 
adaptive decision aids can be an effective instructional tool for behavior 
change and performance improvement (May, Crooks, and Freedy, 1976). Adap- 
tive decision aiding has also been effectively used to show that when 
decision models based on real operators are used for decision making, 
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performance is superior to that of the operator himself, both in terms of 
performance scores and in terms of decision consistency (Dawes, 1970; Freedy, 
Davis, Steeb, Samet, and Gardiner, 1976). 


TECHNICAL APPROACH 


GENERAL. Attention has been initially focused on a single military system 
able to provide good examples of decision making tasks which are important 
to system operation and which are likely to be improved by training in an 
instructional program. This system is the Navy SH-2F (LAMPS) ASW heli- 
copter. An instructional course for the LAMPS crew members (Pilot/ATO 

and Sensor Operator) has recently been developed by Courseware, Inc., San 
Diego. In effect, Perceptronics acted as "subject matter expert" in the 
area of decision making. The final output of the decision analysis, as 
presented in this report, is in a form suitable for incorporation into the 
specification of instructional objectives, the planning of strategies, the 
selection of media, and other other steps required for the production of 
instruction. 


The main tasks involved in the current program effort are the 
following: 


a. Analysis of LAMPS Operators' Functions 
b. Decision Selection 
c. Description of Decision Tasks 
d. Recommendation of Decision Aids 
These tasks are described separately in the following sections. 
ANALYSIS OF LAMPS OPERATORS' FUNCTIONS. An analysis of LAMPS helicopter 
operators identified decision points that occur during execution of the 
operator's functions. This task corresponds to the Job Analysis phase of 
the instructional development process. Decision points in an operator's 


sequence of functions have the following characteristics: 


a. The operator is required to make a choice between a number 
of alternatives. 


b. The criteria for selecting among the alternatives are not 
completely specified, and may have significant judgmental 
components. 


c. The operator is required to either define the possible choices 
(problem structuring) or evaluate the potential outcomes 
(action selection). 


Indicators for identifying decision points in the operator's functions 
include the following: 


a. Additional information may be necessary before a choice can 
be completely defined. 
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b. The possible outcomes of the choice are uncertain and some 
estimate of their likelihood may be required. 


C. The consequences of the actions are composed of a number of 
factors that must be aggregated. 


d. The alternative consequences of the current action will 
differentially affect future actions. 


DECISION SELECTION. From the list of decisions identified in the job 
analysis, a limited number were selected for the examination of decision 
training methodology. Selection focused on highly critical decision. The 
criteria for selection were the following: 


a. Safety Criticality 

b. Time Criticality 

c. Frequency of Occurrence 

d. Current Decision Making Effectiveness 

e. Tractability 

f. Demonstrability 
Candidate lists of potential decision tasks in the LAMPS emergency procedures 
were ranked by LAMPS expert instructors according to the above criteria. A 
composite ranking was then obtained and provided the basis for final decision 
task selection. The results of this selection are presented in Chapter 2. 
The decisions were thus chosen to represent both general classes of decisions 
and decisions that are specific to particular crew positions. 
DESCRIPTION OF DECISION TASKS. This corresponds to the Task Analysis phase 
of Instructional Systems Development. Researchers in decision making have 
suggested various classifications of tasks that decision makers perform 
(Adelson, 1961; Drucker, 1967; Edwards, 1965; Hill and Martin 1971; Howard, 
1968; and Schrenk, 1969). These classifications provide a framework for 
enumerating the tasks that the LAMPS operators must perform in each of the 
selected decisions. A preliminary list of subprocesses involved in gener- 
alized decision making include: 

a. Recognize the decision point 

b. Formulate the alternative courses of action 

c. Establish the possible outcomes of each course 


d. Estimate the multi-attribute utility (i.e., gains or losses) 
associated with each outcome 


e. Estimate the probability of achieving the various outcomes 
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f. Apply decision rule(s) or criteria 
g. Select a best course of action 


A detailed analysis of the selected decision tasks were performed in 
the second phase of the project. The set of decision tasks includes both 
generalized tasks (i.e., decision problem structuring) and tasks that are 
unique to each specific decision (i.e., list of possible alternative 
classes). The results of this analysis appears in Chapter 3. The decision 
trees developed in this phase are included in the appendices of this report. 


DEVELOPMENT OF INSTRUCTIONAL GUIDELINES. The steps involved in the actual 
production of training materials for decision tasks include the specifica- 
tion of: 


a. Objective Hierarchies 
b. Strategy Planning 

c. Media/Method Selection 
d. Course Sequencing 


It is realized that training crew members for more effective decision 
making is no trivial matter. Many years of research have been spent on 
each of the subprocesses involved in decision making. Much work has been 
devoted to the area of probability assessment alone. Nevertheless, it 
has been realized that the effort in analysis of requirements and method- 
ology for decision training in operational systems is justified by the 
following reasons: 


a. The vast amount of theoretical and experimental work provides 
an invaluable guide to practical application. 


b. It appears that much of the benefits to be derived from the 
decision analytic approach lies in problem definition itself, 
that is, in the operator recognizing that a decision problem 
exists, and evaluating it in terms of expected costs and 
expected gains. This aspect of decision making, concepts and 
basic rules, seems highly amenable to training. 


c. By restricting initial effort to a specific system, the job of 
defining decision problems and prescribing procedures for train- 
ing is simplified. 


d. The potential payoff is great. Tactical decisions involve 
equipment and procedures of even-increasing cost. Any gain in 
decision effectiveness realized at the training stage will 
have a large multiplier effect over the total system life. 


A system for generating guidelines to aid development of instructional 
materials for each decision task is developed in this report. This system 
provides essential information to be incorporated in the production of 
instructional materials. Such information includes: 
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a. Identification of decision tasks in objective hierarchies 
b. Classification of decision tasks 
Ce Identification of task-specific and basic instructional contents 
d. Selection of the most effective instructional method 
e. Selection of the most effective instructional media 


RECOMMENDATIONS. OF DECISION AIDS. Identification of decision points and 
decision tasks suggests decision aids that could be incorporated in the 
operational system. Decision aids can be used in a variety of ways to 
improve the logical soundness and the speed of decision making. In general, 
aids are directed toward eliminating deficiencies and limitations in human 
cognitive performance. As an example of eliminating deficiencies, an aid 
can be used to remind a decision maker to seek more information or to 

help avoid a stereotyped response. As an example of overcoming limitations, 
decisions aids can help evaluate a larger number of alternatives than an 
unaided decision maker could consider. A variety of decision aiding 
techniques are available, ranging from printed decision trees and flow 
diagrams, through paper-and-pencil mathematical procedures, to interactive 
computer-based techniques. The computer-based techniques include data- 

base retrieval and bookkeeping functions, graphical problem representation, 
hypothesis generation, simulation of outcome sequences, etc. Operational 

- decision aids have implications for training, since (1) they would be 
expected to reduce or change the elements of the identified decision tasks, 
and (2) the operators would also require training on the use of any software 
or hardware aids introduced to the operational system. 
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SECTION II 
JOB ANALYSIS 
GENERAL 


Since the objective of the program was to develop a methodology for 
the development of decision training programs in specific areas, a task- 
specific appraoch for the research and development program was employed. 
The project started with an analysis of tasks involved in the operator's 
job and constituted the "Job Analysis" phase. 


During the "Job Analysis" phase, the attention was focused on identi- 
fication and selection of decision tasks in the SH-2F (LAMPS) ASW operation. 
An in-depth study of the entire LAMPS operation was conducted. Figure 1] 
presents a detailed diagram of the approach followed during this phase. 
The following is a summary of this approach in which the individual items 
are numerically indicated in the figure. 


a. A preliminary methodology for the identification and classi- 
fication of decision tasks was developed. This methodology 
enables the separation of decision-related LAMPS tasks from 
procedural non-decision tasks. 


b. Two volumes of LAMPS instructional "objective hierarchies" 
were analyzed for the purpose of isolating potential areas 
for decision tasks. These two volumes taken together comprise 
the document: "Instructional Design and Development Objective 
Hierarchies for Pilot/ATO and Sensor Courses," SH-2F (LAMPS) 
Instructional Systems Development Project No. N68221-75-PR- 
$1201, Data Item No. AQO1, prepared by Courseware, Inc., 
7851 Mission Center Court, Suite 220, San Diego, CA 92108. 


Cc. The result of the objective hierarchy analysis was a list of 
potential areas for decision tasks. 


d. An in-depth analysis of the available instructional materials 
for the LAMPS training course was conducted. This consisted 
of the review of 49 Pilot/ATO tape slide courses, 34 crewmen 
tape slide courses, and 187 Pilot/ATO workbook courses. 


e. The result of the instructional materials analysis was a second 
list of potential areas for decision tasks in the LAMPS 
operations. 


Ts From the two preliminary decision area lists, composite 
detailed list of decision tasks was compiled. 


g. To prepare for final decision task ranking and selection, two 


experienced LAMPS instructors and two student LAMPS trainees 
were interviewed with prepared questionnaires. 
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Figure 1. Work Flow For Job Analysis Phase 
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h. The results of the questionnaires provided: 


a ranking of the tentative decision tasks in order of 
importance and criticality, 


An identification of major decisions in each potential area, 
and 


criteria for final selection. 


i. A final composite decision task ranking and selection was made 
based upon all of the analysis information. 


METHODOLOGY 


The functions of both the Pilot/ATO and the Sensor Operator were the 
subjects of the analysis performed in the first phase of the project. A 
preliminary analysis of the job situation specified the required steps in 
the technical approach. These steps include the definition of decision 
tasks, classification of decision tasks, and identification of decision 
task areas. 


Performance of an operator in a decision/non-decision task-pool envi- 
ronment obeys the scheme presented in Figure 2. As the input to this 
scheme, a set of tasks is introduced to the operator as part of the normal 
LAMPS operational procedures. The set contains both decision and non- 
decision tasks. Since processing of a decision task requires a procedure 
considerably different from the one for non-decision tasks, the first 
responsibility of the operator is to identify the task as either a decision 
or a non-decision. Block 1 (Figure 2) shows such a dichotomization. This 
block acts as a "filter" which identifies decision tasks and passes them 
to block 3 for classification. 


Non-decision tasks will also be identified in block 1 and passed to 
block 2 for processing. It is assumed that block 2 contains a strict set 
of predefined procedures for processing non-decision tasks. Instructional 
objectives, outlines and materials for these procedures have already been 
developed by Courseware, Inc., as a part of the current LAMPS pilot 
training course. 


The tasks identified as decision-related are input to block 3, where 
they are classified as Type 1, 2, or 3. Processing of a Type 1 decision 
task requires consideration of problem structuring; a Type 2 decision task 
represents action selection. Type 3 decisions represent a combination of 
both Type 1 and Type 2 in which both problem structuring and action selec- 
tion must be considered. Blocks 1 and 3 form part of the methodology to 
identify and classify decision tasks in the job analysis phase. The 
methodology will be described in the following section. 


DECISION TASK DEFINITION AND CLASSIFICATION. The mechanism of decision 

task identification is similar to that of a "filter" which passes all, 

and only, decision tasks. The characteristics of such a filter are described 
by the definition of a decision task: 
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a. The objective of a decision task is to select an alternative 
from a specified set of alternatives. 


b. This selection may require the formulation of alternatives 
(problem structuring). 


C. There is a lack of completely specified criteria for either 
alternative formulation or alternative selection. 


The operator actions necessary for processing a generalized decision 
task may involve considerations such as establishing possible outcomes and 
consequences of each alternative, determining utilities and probabilities 
of the various outcomes, evaluating major attributes of each available 
alternative, and applying established decision rules for selecting the 
best course of action. Some of these operator actions are shown in Figure 
3 with their relationships to problem structuring and alternative selection. 
In general, the actions of alternative formulation and outcome formulation 
are related to problem structuring while actions such as utility and proba- 
bility assessment are related to the action selection process. 


Since there is a considerable distinction between the two tasks of 
problem structuring and alternative selection, there is a plausible class- 
ification scheme for decision tasks. Such a scheme is created by repre- 
senting a boundary between the decision tasks requiring problem structuring 
and the ones requiring alternative selection. The boundary is defined 
by the types of operator actions necessary to process the decision task. 

It is this distinction which is used to identify and classify specific tasks 
in the LAMPS operational procedures. 


DECISION TASK SELECTION. Once the decision tasks were identified, a com- 
posite ranking method was used to select the final LAMPS decision task 
areas to be used for in-depth study. Preliminary lists of potential 
decision task areas were initially ranked by LAMPS expert instructors 
according to the following criteria: 

a. safety criticality 

b. time criticality 

Cc. frequency of occurrence 

d. current decision making effectiveness 


Perceptronics' analysts then ranked the same preliminary list according to 
the following criteria: 


a. tractability 
b. demonstrability 


A composite ranking was then obtained and served as the basis for final 
decision task selection. 
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INFORMATION SOURCES 


The main information sources used in the job analysis phase originated 
from three main categories: 


a. Objectives Hierarchies 
b. Instructional Materials 
Cc. LAMPS Instructor Interviews 


Objective hierarchies for Pilot/ATO as well as Sensor Operator courses 
were used as the starting point. They consist of a complete organization of 
all LAMPS helicopter operations and are used as a basis for creating train- 
ing courses. The result gained from the analysis of objectives hierarchies 
led to promising LAMPS decision task areas. Among instructional materials, 
Pilot/ATO workbooks were the subjects of detailed studies. The tape slides 
and workbooks deal with specific training subjects and are presented in an 
informal manner conducive to efficient training. LAMPS instructor inter- 
views provided the required information not existing in the previous two 
sources. Potentials, limitations, and efficiency of training elements of 
the present instructional system were part of the information data captured 
through these interviews. Besides the three major sources of information, 
informal interviews of LAMPS student trainees and inspection of actual 
LAMPS helicopters provided additional data for the job analysis. Results 
from each of these information sources is presented in the succeeding 
sections. A detailed listing of decision tasks appears in the appendices. 


OBJECTIVES HIERARCHIES 


Instructional design and development of SH-2F LAMPS operation has been 
represented thoroughly in Pilot/ATO and Sensor Operator Objectives Hierar- 
chies. The objectives are structured in expanding levels of detail with 
each objective decomposed into component sub-objectives. Development of 
instructional materials for courses as well as course sequence are based on 
these hierarchies. However, every element of the most detailed level of the 
hierarchical does not necessarily correspond to a single course. A course 
of instruction may cover many training objectives at different hiearachical 
levels. An example of objectives hierarchies is shown in Figure 4. In some 
cases, aS many as 11 levels of objectives are given. The sample shown in 
the figure is only the top-most level. 


The objectives hierarchies were a primary subject of analysis. This 
methodology for decision task identification was used to identify those ob- 
jectives which could be classified as potential decision tasks. A complete 
list of results of this analysis of Pilot/ATO and Sensor Operator appear in 
tables in Appendices A and B. In these tables, every identified decision 
task is classified as Type 1, Type 2, or Type 3. The tasks are organized 
into related groups in which a problem/sub-problem relationship exists 
between consecutive pairs of elements. The first element of each group 
represents the decision task captured at a higher level. Only the task 
area for the first element of every group is identified since the other 
elements of the group consist of the subproblems of the first element. 
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Perform all Pilot/ATO functions flying an SH-2F. 
Fly an SH-2F to complete an ASW mission. 
Fly an SH-2F to complete an ASMD mission. 


Fly an SH-2F to complete a search and rescue mission. 


Fly an SH-2F to complete an external cargo mission. 


1.0 
1.1 
1.2 
1.3. Fly an SH-2F to complete a surveillance mission. 
1.4 
1.5 
1.6 


Fly an SH-2F to complete an Internal cargo/passenger 
transfer mission. 


1.7. Fly an SH-2F to complete a Naval gunfire spotting 
mission. 

1.8 Describe the main components and operation of SH-2F 
aircraft systems. 


Figure 4. Objectives Hierarchies 
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The identification and classification process for decision task 
followed the methodology explained earlier. Due to the large number of 
existing tasks, the plausibility of computerization of such an analysis 
was considered. A correspondence between the decision-related tasks in 
the objectives hierarchies and certain "keywords" used in the description 
was noted during the analysis. A keyword, in this case, refers to an element 
of a set of verbs appearing in an imperative sentence, for example, "Deter- 
mine whether to land the aircraft or to abort the recovery" (p. 112, Pilot/ 
ATO Objective Hierarchies) or "Perform a search in accordance with NWP-37 
and NATOPS" (p. 130 Pilot/ATO Objective Hierarchies). The description 
of every task appearing in objectives hierarchies includes at least one 
of these keywords. 


The result of examination of objectives hierarchies, however, revealed 
an inconsistency in the use of keywords. Some critical keywords appeared . 
in both decision tasks as well as non-decision tasks. For example, "state" 
and "calculate" appeared mainly in non-decision tasks; however, about 20% 
of their applications were in definitions of decision tasks. Future con- 
sistency in the use of keywords for decision and non-decision tasks is 
strongly recommended. A list of the two classes of keywords appears in 
Table 1. In spite of the inconsistency in use, the keywords provided a 
reasonable starting point for identification of decision tasks. The use 
of a computer program for this part of decision task identification could 
be a means of increasing the efficiency of identification process. 


In summary, the analysis of the objectives hierarchies led to the 
conclusion that most of the decision tasks would be found in the emergency 
and tactical operational procedures. Furthermore, the Pilot/ATO tasks 
contained many more decision-related situations than the sensor operator 
tasks. 


INSTRUCTIONAL MATERIALS 


Analysis of objectives hierarchies identified the major areas for 
decision making tasks. In this section of the job analysis, the study was 
focused on the instructional materials. Two out of four existing types of 
course materials have the major role in the instruction of LAMPS operation. 


a. Tape Slides (TS) 
b. Work Book (WB) 


c. Computer Aided Instruction (CAI) 
(not included in the analysis due to unavailability) 


d. Visual Tapes (VT) 
(not included in the analysis due to unavailability) 


TAPE SLIDES. A series of audio cassette tapes together with corresponding 
synchronized slides are used to cover part of the course instruction in 
LAMPS operation. Tape slides for Pilot/ATO, as well as crewmen, were the 
subject of a detailed analysis. Tape slide materials mainly represent 
technical and introductory courses. The majority of tape slide courses 
cover the procedural tasks with no decision making steps. These were 
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TABLE 1. 


OBJECTIVES HIERARCHY KEYWORDS 


DECISION MAKING KEYWORDS 


DETERMINE 
CALCULATE 
STATE 
PRESCRIBE 
IDENTIFY 
CLASSIFY 
ESTIMATE 


INTERPRET 
REVISE 
SELECT 
INDICATE 
RECOGNIZE 


NON-DECISION MAKING KEYWORDS 


CALCULATE 
PERFORM 
RECOVER 
PLAN 
CONDUCT 
DEMONSTRATE 


LIST 
NAME 
DESCRIBE 
STATE 
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termed Non-Decision Tasks (ND). Some of the courses represented by tape 
slides covered decision situations in which the pilot has the full respon- 
sibility of decision making (Decision Making Tasks, DM). In the remainder, 
the decision making tasks were assumed to be performed by the pilot through 
the execution of a predefined procedure. These were termed Decision 
Execution Tasks (DE). 


In some cases, the decision making characteristics of a particular 
situation are explicitly emphasized in the tape slide presentation. The 
trainee is told that a decision task exists and is given situations to 
analyze. However, it is made exceptionally clear the the final choice of 
action rests with him (the pilot or the crewman) and that the basis for 
the decision is his own judgment and experience. These decision tasks are 
"obvious" in the sense that they are termed “decision tasks" in the in- 
structional materials themselves. In other cases, a decision task is 
evident in the instructional materials but is not presented to the student 
as such. These decision tasks are "hidden" within established procedures 
for executing a particular operational plan. 


The results of tape slide analysis for 49 Pilot/ATO and 34 Crewmen 
courses appear in Appendices C and D. Task number, type, and keyword columns 
are identical to the ones of objectives hierarchies analysis. The task 
name identifies the course title and the slide number represents the new 
codes replacing the previous coding system to identify different tape 
slides. The non-available tape slides are marked by NA. Some of the tasks 
are described in the notes following the result of the analysis. These 
tasks are marked by an asterisk on their slide numbers. 


WORKBOOKS. A considerable portion of the LAMPS operational instructions are 
covered by workbooks. Figure 5(a) through 6(b) shows examples of work- 

book courses for a "Radar Offset" approach to an unidentified contact and 
procedures for an "Immediate Ditching" of the helicopter, respectively. 

In addition to the exercise and lesson number, the first page of a work- 
book course always contains the objective, a general list of actions, and 

an introduction describing the course material. The remainder of the 

lesson is usually divided into procedures, actions, and remarks or comments. 


In Figure 5(b), the Radar Offset Approach lesson, the action performed 
for procedure 2 is "The ATO will decide if a direct or offset approach will 
be used." Guidelines for this decision are then given. This is an example 
of a Type 2 "obvious" decision in which the ATO must choose from among a 
specified set of two alternatives. However, Figure 6(b), the Immediate 
Ditching lesson, under Section 4.B.2 shows a hidden decision in which the 
pilot is not given guidelines or told how to choose an alternative. 


The analysis of the Pilot/ATO workbooks included 187 lessons. The 
results of this analysis appear in Appendix E. However, a summary of 
results appear in Table 2. © 


EXPERT INTERVIEW 

The information acquisition for the job analysis phase was completed 
by interviewing the LAMPS operational experts. Two LAMPS operation instruc- 
tors as well as two LAMPS operation trainees participated in the interviews. 
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When flying a radar approach to a contact the senso will provide 
headings and distances to fly. You may deceive the contact into 
thinking that he has not been detected by flying an offset approach. 
(An offset approach entails flying parallel or perpendicular to the 
contact’s course and then “cutting in’ directly to him.) If you 
believe the contact will submerge before you reach him, a straight 
in approach is recammended. (It is easier to calculate the estimated 
time on top flying a straight in approach.) 


Figure 5(a) 


30 


J2 


NAVTRAEQUIPCEN 77-C-0005-1 


RADAR OFFSET | 


Ey Saarinen ag 
The senso should locate the contact 


and pass the bearing and range 
information to the cockpit. 


PROCEDURE 


reas 


FIND RADAR 


The ATO wil) dec 
"off 


DETERMINE IF 


ADIRECTOR | offset approach will be used, 
APPROACH ; ; ; 
Wiens apian : a. Adirect approach is flown directly a. This approach is recommended if 


from the aircraft's present position 
to the surface contact. 


you believe you have been datected. 


STRAIGHT IN sy CONTACT 
AIRCRAFT 


b. If you think you are detected and 
wish to get closer to the contact 
prior to starting your direct run, 
you should use the offset appraach. 
If the contact disappears from the 
radar during an offset approach,- 
immediately perform a direct 
approach to the contact. 


CLOSEST POINT OF APPROACH 


b. The offset approach is flown toa 
position abeam the target to one 
side at the closest point of 
approach, From this point a 

direct leg is used. 


CONTACT 
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Determine the distance from the 
aircraft to the contact and adjust 
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Time on top is important in case your 
contact disappears from the radar 
screen. 


Figure 5(b) 
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IMMEDIATE DITCHING (PILOT) 
vk 
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use the call position of the ICS to 
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Figure 6(b) 
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TABLE 2. RESULTS OF INSTRUCTIONAL MATERIAL ANALYSIS 


OBVIOUS DECISIONS 


External Engine Fire (TS) 

Gear Box 0i1 Malfunction (TS) 
Determining Restart Feasibility (TS) 
Radar Offset Approach (WB) 


HIDDEN DECISIONS 


Combining Gear Box 011 Malfunction (TS) 
Hydraulic System Malfunction (TS) 
Electrically-Caused Overspeed (TS) 

Loss of Tail Rotor Thrust (TS) 

Loss of Tail Rotor Control (TS) 


Immediate Ditching (WB) 
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A series of predesigned questionnaires was prepared from the specific tasks 
identified in objectives hierarchies and tape slide analysis. A sample 
questionnaire appears in Table 3. The questionnaire concerns the problem 
of a gearbox malfunction. It is classed as an emergency operation. Each 
question attempts to elicit information about the particular decision. 


Question Q1 asks for a set of alternative actions which can be taken 
when a gearbox malfunction occurs. Q2 tries to locate attributes of each 
alternative which may be used in selection. Since limiting the initial 
set of alternatives can be a help in rapid decision making, question Q3 
probes for situations which allow the deletion of one or more alternatives. 
Questions Q4 and Q5 request value judgments on relative utility and proba- 
bility of occurrence, respectively. Only a linguistic value judgment is 
required in this case. Normally, however, numerical judgments are preferred. 
Q6 is a structural question. It determines if this is a "one shot" 
decision situation or if it is repetitive in nature. Questions Q7, Q8, and 
Q9 concern the "cues and alerts" for a gearbox malfunction. These are 
indicators that a malfunction is either about to happen or has already 
occurred. Question Q8 deals with completeness. That is, if there is an 
imminent gearbox malfunction, will all cues and alerts be observable? 
Question Q9, on the other hand, deals with uniqueness. That is, are there 
any other malfunctions with exactly the same cues and alerts as a gearbox 
malfunction? If this situation could occur, there would exist an ambiguity 
in the recognition of an emergency decision situation. In such cases, it 
would be beneficial to design a new alert (such as an emergency indicator 
light) to differentiate between the two ambiguous conditions. 


Questions Q10, Q11, and Q12 produced the most valuable information 
for the selection of decision task areas for further study. Q10 required 
the participants to rank the gearbox malfunction with respect to other 
potentially critical areas. The only criteria given was "importance" 
and it was left to the participant to define this word in his own terms. 
Immediately after this ranking, the participant was asked to describe the 
major factors that he considered when thinking about "importance." Both 
instructors and both students gave the same list of criteria: 


a. Personnel Safety 

b. Mission Effectiveness 

c. Equipment Salvage 
This unanimous consensus is, of course, attributable to the fact that all 
participants were part of the same training program. However, all agreed 
that not only were these the most important, but also that there were no 
other major ones. 

The last question, Q12, required the ranking of the potential decision 


areas in terms of each criteria separately. The detailed results of these 
rankings will be described in the next sections. 
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TABLE 3. SAMPLE OF EXPERT INTERVIEW QUESTIONNAIRE 


WHAT MAY BE DONE WHEN A GEARBOX MALFUNCTION HAPPENS? 


WHAT ARE THE ADVANTAGES AND DISADVANTAGES INVOLVED IN 
EACH ALTERNATIVE? 


IS THERE ANY SITUATION IN WHICH SOME OF THE ALTERNATIVES 
ARE USELESS? WHICH ONE AND IN WHAT SITUATION? 


ARE ANY OF THE ALTERNATIVES MORE PREFERABLE IN CERTAIN 
SITUATIONS? WHICH ONE AND IN WHAT SITUATION? 


WHAT IS THE CHANCE OF SUCCESS FOR EACH ALTERNATIVE? 
(GOOD, BAD, NOT LIKELY, ...) 


IF THE ATTEMPT TO PERFORM AN ALTERNATIVE FAILS, WILL 
THERE BE A CHANCE TO TRY ANOTHER? HOW? 


WHAT ARE THE CUES AND ALERTS FOR SUCH A MALFUNCTION? 
DO ALL CUES EXIST IN CASE OF THIS MALFUNCTION? 

IS THERE ANY OTHER MALFUNCTION WITH THE SAME CUES? 

HOW DO YOU RANK THE IMPORTANCE OF SUCH A MALFUNCTION? 
WHAT WAS THE CRITERIA FOR IMPORTANCE FOR THIS RANKING? 
RANK THE TASK FOR EACH CRITERION SEPARATELY. 


36 


NAVTRAEQUIPCEN 77-C-0005-1 
CANDIDATE DECISION TASKS 


Analysis of instructional materials identified the decision task areas. 
This information, together with the potential decision task areas detected 
in objectives hierarchies analysis (and confirmed in expert interviews), 
produce the list of final Pilot/ATO decision tasks. The list shown in 
Table 4 contains fifteen decision task areas covering a variety of oper- 
ational procedures. The list formed the basis for the selection of two 
major areas for in-depth analysis in the next phase. 


SELECTION CRITERIA AND ANALYSIS 

Two classes of criteria were identified for the final selection of 
decision tasks. The first class contains the criteria of importance as 
judged by LAMPS/ASW operational experts. This class includes: 

a. Safety criticality | 

b. Time criticality 

c. Frequency of occurrence 

d. Current decision making effectiveness 

The second class concerns criteria for usefulness of the task to demon- 
strate the methodology of decision training. The two criteria of this 
class were: 

a.  Tractability 

b.  Demonstrability 
where tractability is an aggregation of three basic components: 

a. Representativeness 

b. Size 

c. Degree of Abstraction 

Since all the criteria were equally important to the object of the task 
selection, an equal-weight (w=1) utility model was used to make the final 
selection. A weight of w=-1 was given for "current decision making effec- 
tiveness" since this reflects the negative effect of this criteria on the 
task selection. 

The detailed analysis chart for the decision task selection appears in 
Table 5. The entries of each column are the result of the ranking of the 
candidate decision tasks by the interview participants with respect to the 
corresponding criteria. The last column, Mean Ranks, is an indication of 


priority for selection of each task according to the above criteria and 
evaluation scheme. 
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TABLE 4. FINAL CANDIDATE PILOT ATO DECISION TASKS 


FLYING METHOD (0H) . ENGINE QUITS (0H) 

MISSION SAFETY (II) . EQUIPMENT MALFUNCTION (OH) 

TYPE OF LANDING (II) . GEARBOX MALFUNCTION (TS) 

LAND OR WAVE OFF (II) . HYDRAULIC SYSTEM MALFUNCTION (TS) 


ABORT OR CONTINUE THE MISSION (OH) . LOSS OF ROTOR THRUST (TS) 
DITCHING SITUATION (WB) . LOSS OF TAIL CONTROL (TS) 
RESTART FEASIBILITY . FIRE (TS) 

RADAR APPROACH (WB) 


OH: OBJECTIVE HIERARCHY 
II: INSTRUCTOR INTERVIEW 
WB: WORK BOOK 

TS: TAPE SLIDE 
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TABLE 5. ANALYSIS OF DECISION TASK SELECTION 


Frequency 


ae Safety Time of Current DM 74 a73 mean 
Nect stan: Task Criticality Criticality Occurrence Effectiveness Tractability Demonstrability Ranks 
(w=1) (w=1) (w=1) (w=-1) (w=1) (w=1) 

Flying Method 14 12 9 4 7 8 46 
Mission Safety 13 1 8 5 15 15 57 
Type of Landing 12 13 1 3 6 7 35 
Land or Wave Off 9 10 2 6 4 3 22 
Abort or Continue the Mission a 7 4 9 3 4 35 
Ditching Situation 5 8 15 8 1 ] 22 
Restart Feasibility 7 9 13 7 8 5 35 
Radar Approach 15 15 5 ] 13 6 53 
Engine Quits 1 3 12 130 2 2 7 
Equipment Malfunction 10 14 3 2 5 9 39 
Gearbox Malfunction 4 4 7 12 10 am 24 
Hydraulic System Malfunction 8 5 6 11 11 12 31 
Loss of Rotor Thrust 2 ] 14 15 9 10 21 
Loss of Tail Control 6 6 YW 10 12 14 39 
Fire 3 2 10 14 14 13 28 


(w=weight) 
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The ranks in each column were obtained as follows. Three participants 
independently ranked the tasks under safety criticality. Two of these 
rankings were almost identical and one varied greatly. The ranking which 
was at variance was disregarded, and the other two were averaged and re- 
ranked. The three criteria, Time Criticality, Frequency of Occurrence, 
and Current Decision Making (DM) Effectiveness were ranked by one par- 
ticipant each. The two criteria, Tractability and Demonstrability, were 
ranked by Perceptronics' analysts before the interviews. 


RECOMMENDED DECISION TASKS 


An additive model with unit weights was used to determine the final 

composite ranking. The numbers in the Mean Ranks column in Table 5 are 

a result of adding together the rank values of each of the criteria columns 
with the exception of Current DM Effectiveness which was subtracted. The 
final ranking is shown in Table 6. "Engine Quits" was by far the most 
important with "Loss of Tail Rotor Thrust," "Land or Wave off," "Ditching 
Situation," and "Gearbox Malfunction" closely grouped. In this group, 
Land or Wave off was chosen since it would provide possible experimentation 
without danger. Loss of Tail Rotor Thrust and Gearbox Malfunction are 
technical tasks similar to Engine Quits. Ditching Situations are a direct 
result of Engine Quits and could benefit from this analysis. Table 7 
presents characteristics of the two selected decision tasks and compares 
and contrasts them. The selection decision tasks for in-depth analysis: 
Engine Quits and Land orWave off, represent a rich environment for appli- 
cation of decision training methodology. 
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TABLE 6. SELECTION ANALYSIS RESULTS AND RECOMMENDATION 


ENGINE QUITS 

LOSS OF ROTOR THRUST 

LAND OR WAVE OFF 

DITCH SITUATION 

GEARBOX MALFUNCTION 

FIRE 

HYDRAULIC SYSTEM MALFUNCTION 


*Recommended Decision Tasks 


4] 


RESTART. FEASIBILITY 

ABORT OR CONTINUE THE MISSION 
TYPE OF LANDING 

EQUIPMENT MALFUNCTION 

LOSS OF TAIL CONTROL 

FLYING METHODS 

RADAR APPROACH 

MISSION SAFETY 


A: 
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TABLE 7. CHARACTERISTICS OF RECOMMENDED DECISION TASKS 


TASK 


ENGINE QUITS 


LAND OR WAVE OFF 


CHARACTERISTICS 


VERY HIGH SAFETY CRITICAL, VERY HIGH TIME 
CRITICAL, VERY LOW FREQUENCY OF OCCURRENCE, 
VERY LOW CURRENT DECISION MAKING EFFECTIVENESS, 
VERY GOOD DECISION TASK REPRESENTATIVE, VERY 
LARGE PROBLEM SIZE, LOW DEGREE OF ABSTRACTION, 
VERY HIGH DEGREE OF DEMONSTRABILITY, TECHNICAL 


MEDIUM SAFETY CRITICALITY, LOW TIME CRITICALITY, 
VERY HIGH FREQUENCY OF OCCURRENCE, MEDIUM 
CURRENT DECISION MAKING EFFECTIVENESS, VERY 

GOOD DECISION TASK REPRESENTATIVE, SMALL 
PROBLEM SIZE, MEDIUM DEGREE OF ABSTRACTION, 

VERY HIGH DEGREE OF DEMONSTRABILITY, 

OPERATIONAL 
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SECTION III 
DECISION TASK ANALYSIS 
GENERAL 


To identify the characteristics of the decisions involved in each 
decision situation, a detailed decision task analysis was performed on the 
decision situations selected in the first phase. This analysis constituted 
the second phase of the project and provided detailed task specific infor- 
mation. This information may be regarded as secondary instructional con- 
tent for the objective decision training program. Although the secondary 
instructional content is not vital to the training program, its inclusion 
may be desirable in some cases. The decision of whether to include the 
secondary instructional content will be made at the time of actual in- 
structional material development by the instruction technologist and 
based on the desired level of detail and degree of task-dependency for the 
objective training program. 


During the second phase of the contract, the "Decision Task Analysis" 
phase, a detailed analysis of the selected decision task areas was per- 
formed. The results of this analysis led the program into two groups of 
activities. The first group consisted of further research activities in 
the task areas in order to identify the characteristics of the decision 
tasks involved as well as the required facts and elements contributing 
to the decision making process. The second group involved theoretical 
analysis performed to reveal the underlying concepts, relevant decision 
rules, and optimal procedures for the process. The first group provides 
task-specific knowledge while the second provides the theoretical backup 
as well as domain-independent data required for the development of the 
guidelines for instructional materials design. 


Figure 7 shows the work flow diagram of the Decision Task Analysis 
Phase. The following is a summary of the work accomplished during this 
period. The individual items are numerically indicated in Figure 7. 
Detailed description of major activities are presented in succeeding 
sections. 


a. A detailed analysis of the selected decision task areas was 
performed. A complete study of the workbook materials as well 
as tape slides relating to the task areas supplied the formal 
data, which, together with the expert interview results, 
provided the background knowledge in the selected tasks. 

Since ditching was recognized as a critical task area related 
to engine quits, it was analyzed in this activity in addition 
to the two selected areas. 


b. The decision elements involved in each decision task area were 
identified. The alternative courses of action for each decision 
and the possible outcomes for each action were specified. 


c. Based on the identified decision elements, the decision trees 
for engine quits, land or wave off, and ditching decision task 
areas were constructed. 
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A set of questionnaires and a number of scenarios were designed 
to be used for examination, completion, and possible modification 
of the generated decision trees. The questionnaires are designed 
to provide the probabilities and utilities on the decision trees. 
The scenarios are designed to test the structure of the decision 
trees and acquire possible additional alternatives for decision 
tasks through a protocol analysis. 


The developed scenarios were used in a protocol analysis to test 
the structure of the decision trees and acquire possible additional 
alternatives for decision tasks. LAMPS operation instructors were 
the participants of the protocol analysis. 


The designed questionnaires were used to provide the probabilities 
and utilities on the decision trees. This information was 
acquired through a LAMPS operation expert interview. 


Based on complete decision trees, all decision tasks involved in 
the selected decision task areas were identified. A complete 
decision solution for each one was directly produced from the 
task area decision trees. 


Different decision making methods and procedures were analyzed 
and the ones suitable for identified decision tasks were selected. 
The analysis was heavily based on the domain-dependent knowiedge 
acquired in the preceding activities to provide the task-specific 
knowledge required for development of the guidelines for instruc- 
tional material design. 


Two different roles for a human operator as a decision element 
were identified: an operator as a decision maker and an oper- 
ator as a decision executor. 


A visit to the Los Angeles Police Academy was made to review the 
"shoot/no shoot" program. The program is a decision training 
methodology by simulation focusing on the problem of when to 
shoot rather than how to shoot. Although there is no formal 
training in either probability theory or decision theory, there 
are extensive courses in the established important attributes 
that should be considered when deciding whether or not to shoot 
in a given situation. The application of a similar approach 
enriched with a better understanding of decision making elements 
was studied for the training of time-critical decision tasks 
(see Appendix I). 


Two different approaches, generalized and task-specific decision 
training, were analyzed and the validity of each one for the 
decision training program was studied. In a task-specific 
decision training program, the task-domain knowledge is viewed 
as a nucleus around which dacision training will be built. In 
this case, decision training development starts from the most 
task-dependent level and progresses by addition of less 
task-dependent levels. However, a generalized decision training 
program must be developed without an exact knowledge of the 
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application domain. Therefore, the development of the train- 
ing program must be such that it "contains" all possible appli- 
cation domains. In this approach, the program starts with the 
most general levels of decision training and progresses to the 
ones with a lower degree of generality. 


1. A comprehensive list of words, phases, and terms were classified 
into two groups of decision related and unrelated "behaviors." 
The decision related group provides a list which should be used 
‘to define the instructional objectives of the decision program. 


m. The decision making subprocesses were decomposed into relevant 
behaviors that the trainee must gain by the end of the decision 
training program. The list of relevant behaviors recognized by 
this decomposition will provide a task-independent set of data 
required to construct instructional objectives which, in turn, 
will be used in the development of instructional materials. 


METHODOLOGY 


A series of activites in the second phase of the project involved 
theoretical analysis. The activities include: 


a. Identification of decision maker/decision executor 
b. Identification of decision-related behaviors 
c. Analysis of generalized/task-specific decision training 


d. Decomposition of decision subprocesses into relevant behavioral 
objectives. 


These activities constitute a basis for development of the methodology 
designed for the decision training program. Items (a) and (b) as well as 
an overview of decision tree representation are described in this section. 


DECISION MAKER VERSUS DECISION EXECUTOR. Analysis of the possible roles of 
a human operator as a decision element provides information concerning the 
nature of the training task. Such an analysis was performed, and two 
different roles for a human operator as a decision element were identified: 
an operator as a decision maker and an operator as a decision executor. 

The distinction between the two roles can be defined by the degree of aiding 
that is provided to the operator at the time of interaction with the 
decision problem. If the decision problem has been analyzed prior to its 
actual occurrence and a general solution in terms of a reasonably specified 
procedure has been developed, the responsibility of the operator involved 
with the decision problem is to "execute" the procedure with the consid- 
eration of the problem-environmental conditions. However, if such a 
procedure does not exist, the operator will also be responsible for gen- 
erating the procedure for selecting the decision solution (Figure 8). 

For critical decision tasks, the decision making performance can be ~ 
increased by shifting the role of operator from decision maker to decision 
executor as much as possible. This objective may be partly accomplished 
through training. : 
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CURRENT 
ENVIRONMENTAL 
CONDITION 


Figure 8. Decision Maker/Decision Executor 
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DECISION RELATED BEHAVIORS. The success of any decision training program 
can be evaluated by measuring the degree of improvements in a set of 
desirable behaviors provided by the training program at its conclusion. 
The set of behaviors includes the relevant cognitive abilities required 

to accomplish the goals of the training program: effective decision mak= 
ing in a specific environment. Such a set constitutes the decision 
training behavioral objectives. The first step to identify the decision 
training behavioral objectives is an analysis of different training be- 
havioral terms. Such analysis was performed and decision-related be- 
havioral terms for different classes of training were identified (Table 8). 


DECISION TREE REPRESENTATION. A decision problem and its solution may be 
represented in different formats. Decision trees constitute a structured 
format for such a purpose. Since decision trees provide the most descrip- 
tive means for representation of decision problems, they were used to 
describe the results of the decision task analysis performed for each of 
the selected decision tasks. This method of representation is described 
briefly in this subsection. 


A decision tree is a structural representation of sequences of antic- 
ipated decisions and events in a specific environment. A conventional 
decision tree is composed of three different nodes: decision nodes, 
event nodes, and terminal nodes. A decision node represents the alter- 
native courses of action possible at a situation described by the path 
from the root of the tree to the decision node (normally indicated by 
a small square -- Figure 9a). An event node represents different out- 
comes which might occur as a result of application of a specific alterna- 
tive course of action in a specific situation described by the path from 
the root of the tree to the preceding decision node. An event node 
is normally represented by a small circle (Figure 9b). A probability 
of occurrence is assigned to each branch emerging from an event of the 
corresponding event. The probabilities of events associated with each 
event node must sum to one. That is, the outcomes at each event node 
must be mutually exclusive and exhaustive. A terminal node represents an 
outcome which can be judgmentally assessed with reasonable accuracy. Such 
assessment provides a utility number which is a quantitative evaluation 
of the degree of desirability of the corresponding outcome in the context 
of the specific situation. In some cases, due to the ease of assessment, 
outcomes are assessed in terms of losses involved instead of utilities. 

In such cases, a loss number at each branch represents the degree of 
aversion for the corresponding outcome in the context of the specific 
situation. Terminal nodes are represented by a small triangle (Figure 9c). 


Analysis of different decision tasks revealed the need for the intro- 
duction of a new type of node into decision trees. Such a node, called a 
"situation node," represents possible situations which might be encount- 
ered at a specific point on the decision tree. At the planning stage of 
a decision process, the involved person (the decision maker) has only a 
knowledge concerning all possible situations that may exist. However, 
unlike the case of event nodes, at the actual decision implementation 
time (immediately before the decision) the involved operator (the decision 
executor) will know with certainty which of the possible situations 
actually exists. Therefore, treatment of such situations was an event 
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TABLE 8. DECISION-RELATED BEHAVIORAL TERMS 


General Instructional Objectives General Discriminative Behaviors 
Analyze Evaluate Choose Define 
Apply Recognize Describe Detect 
Compute Think Differentiate Distinguish 
Use Identify Select 

Complex, Logical, Judgmental Laboratory Science Behaviors 

Behavior 
Analyze Compare Apply Calibrate 
Decide Evaluate Conduct - Decrease 
Formulate Generate Increase Plan 
Induce Infer Weigh 
Plan 

Creative Behavior Mathematical Behavior 
‘Generalize Modify Compute Count 
Predict Question Estimate Measure 
Rearrange Verify Calculate 

Miscellaneous 
Consider Determine 
Discover 
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ALTERNATIVE OUTCOME 
(PROBABILITY) 


DECISION NODE EVENT NODE 
(a) (b) 
SITUATION 
aoecmee, AS (UTILITY) 
TERMINAL NODE SITUATION 
NODE 
(c) (d) 


Figure 9. Decision Tree Components 
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node, and assignment of subjective probabilities will not provide a 
successful representation for the problem. Situation nodes are repre- 
sented by a small circle surrounding an "X" (Figure 9d). 


An example of a situation node arises when the "Engine Quits: situa- 
tion is encountered. A decision maker (may be the pilot himself), who is 
analyzing this situation prior to actual occurrence, knows ‘that such a 
situation may be caused by either a single engine failure, a dual engine 
failure, or a fuel starvation. He is also aware of the fact that each 
of these situations are different in nature and, although all may be 
represented by a single "Engine Quits" situation, different courses of 
action must be taken upon the occurrence of different situations. There- 
fore, this event must be divided into three possible situations (Figure 
10). However, in such a case, assignment of subjective probabilities of 
occurrence to the three situations will not be helpful since at the time 
of actual occurrence of this situation, the pilot knows which of the 
three situations has occurred and will proceed on the branch representing 
that situation, regardless of the probabilities. In a way, a situation 
node may be viewed as an event node in which the probabilities of different 
branches emerging from it can not be assessed successfully at the time of 
decision analysis. However, at the actual decision time, just before the 
selection of the course of action, the probability of one of the branches 
becomes one and the rest will be zero. In other words, a simple observa- 
tion will provide a masking scheme for the pilot to ignore the irrelevant 
portions of the tree and concentrate only on the subtree emerging from 
the branch representing the current situation. A simple example of a 
decision tree appears in Figure 10. 


DECISION ELEMENT IDENTIFICATION 


A detailed analysis of the relevant workbooks and tape slides as wel] 
as expert interviews identified the decision elements involved in the 
selected decision task areas. The alternative courses of action for each 
decision and the possible outcomes for each action were specified. The 
result was a large amount of data which was incorporated in the construc- 
tion of preliminary decision trees for the decision task areas. 


DECISION TREE ELICITATION 


Bases on the results of the detailed analysis of the selected decision 
tasks, a preliminary decision tree was constructed for each of the decision 
tasks. These decision trees represent the relevant academic knowledge 
required for decision making in the selected areas. The structured decision 
tree representation provides a suitable perspective for inspecting these 
situations in terms of decision analysis. However, the development of the 
complete decision tree depends on the elicitation of non-academic pieces of 
knowledge from the experts. These pieces of knowledge result from exper- 
jence, and since there is no formal way of teaching such items, they are 
usually not included in training materials. The process of decision tree 
elicitation from experts can capture such knowledge and represent it ina 
formal structure. 


To implement this approach, a set of scenarios and a series of 
questionnaires were developed and used to elicit the complete decision 
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trees. The scenarios were used in protocol analyses to modify the 
structure of the trees, and the questionnaires were utilized in expert 
interviews to provide probabilities of different outcomes as well as the 
utilities of the terminal nodes. The protocol analyses and the expert 
interviews are described in the following subsections. The complete 
decision trees for "engine failure" and "land or wave off" appear in 
Appendices F and G, respectively. 


PROTOCOL ANALYSIS. A protocol analysis was performed to check the structure 
of the preliminary decision tree. A scenario was developed to acquire 
alternative courses of action at each decision node, and possible outcomes 
at each event node. A sample scenario appears in Table 9. The complete 
list of scenarios and the questions to be answered by the protocol analysis 
is included in Appendix H. Four LAMPS ASW instructors were participants 

in the protocol analyses. The results of the analyses provided the 
experience-oriented knowledge which was utilized to modify the structure 

of the preliminary decision trees. 


EXPERT INTERVIEW. Once the structures of the decision trees were completed, 
probabilities of different outcomes and utilities of different terminal 
nodes were judgmentally assessed. To accomplish this objective, four LAMPS 
ASW instructors were interviewed. To simplify the process of eliciting sub- 
jective probabilities the participants were asked to assign a number to 

each outcome such that the number represented the relative chance of 
occurrence of the corresponding outcome. In this case, the participants 
were not obliged to provide probabilities which add up to one. Therefore, 
the elicitation process proceeded more efficiently. Since the protocol 
analyses had provided the exhaustive list of all possible outcomes at each 
event node, a simple normalization procedure, using chances of outcomes 
emerging from each event node, calculated the final probabilities of the 
outcomes. To provide further simplification, the participants were asked 
to order the outcomes emerging from each event node according to their 
chance of occurrence prior to actual assessment of the probabilities. With 
such a procedure, the process of probability assessment was performed in 

an informal manner, and without too much emphasis on the theoretical notion 
of probabilities. Table 10 represents a sample questionnaire for probabil- 
ity assessment. The results of the probability assessment are indicated 

on the decision trees in Appendices F and G. 


Five major elements were identified as basic attributes for the terminal 
outcomes. These include (1) total damage to the aircraft, (2) partial 
damage to the aircraft, (3) loss of pilot life, (4) loss of sensor operator's 
life, and (5) loss incurred from mission failure. At each terminal node, 
an informal multiattribute utility analysis was performed on the five 
attributes. To perform such analysis, the degree of severity of each 
attribute was elicited from each participant. The questionnaire used for 
this process appears in Table 11. In these situations, the assessment of 
the degree of aversion is more natural than the assessment of the degree 
of desirability. Therefore, instead of utilities, the losses incurred by 
different attributes were assessed. The results of this assessment by the 
four participants appear in Table 12. The degree of aversion for each of 
these situations is different in wartime compared to a training environment 
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TABLE 9. SAMPLE SCENARIO FOR PROTOCOL ANALYSIS 


Consider yourself engaged in each of the following 


scenarios. Describe what might happen in terms of 


the possible alternative courses of action, possible 
states of nature, and the outcomes associated with 
each combination of a state of nature and a course 


of action. 


You have a single engine failure 
while flying over land. One engine 
is sufficient for the weight of the 
aircraft and there is a safe site 


in the area. 
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TABLE 10. SAMPLE QUESTIONNAIRE FOR PROBABILITY ASSESSMENT 


We want to know the probability that each outcome occurs. You are asked 
to consider each outcome and tell me how often you think these situations 
occur. For example, in the land or wave off situation, you have decided 
on a running landing: under these conditions, how often are you going 
too fast, how often is the sink rate too high, how often is it OK? 


Please use this scale for each event and try to give me a number to 
represent the relative chance of occurrence of each event. 


0 10 20 30 40 50 60 70 80 90 100 
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TABLE 11. LOSS ASSESSMENT FOR MAJOR OUTCOME ATTRIBUTES 


Determine the value for each of the following components 
of loss on a scale from 0 to 1000. 


1. A/C total damage 

2. A/C partial damage 

3. Loss of pilot's life 

4. Loss of sensor operator's life 


5. Loss incurred from mission failure 


1000 
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TABLE 12. THE RESULTS OF LOSS ASSESSMENT FOR 
THE FIVE MAJOR OUTCOME ATTRIBUTES 


1. Loss of Pilot's Life 
2. Loss of Sensor Operator's Life 
3. Loss of Mission 
a. Major 
b. Minor 
4. A/C Total Damage 
5. A/C Partial Damage 
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in peacetime. Since the participants were the experts in a peace- 

time training environment, the losses were assessed for a peacetime 
condition. The results of the loss assessment also indicated the 
differences among different participants' scales. This information was 
incorporated in the aggregation of the losses assigned by different par- 
ticipants to the same outcome. A sample questionnaire for expert interview 
is presented in Table 13. The results of the loss assessments are indicated 
on the decision trees of Appendices F and G. 


58 


NAVTRAEQUIPCEN 77-C-0005-1 


TABLE 13. SAMPLE QUESTIONNAIRE FOR EXPERT INTERVIEW 


In a training environment (during peacetime ) when one engine is not 
sufficient for the total weight of the aircraft, what is the chance 
that after jettisoning: 


A) The aircraft will be light enough so that one engine 
is sufficient? 

B) The aircraft will not be light enough so that one 
engine is sufficient? 


A dual engine failure occurs when flying over water.. How do you 
assess the losses incurred from each of the following possible 
outcomes on a 0 to 1000 scale: 


A) Loss of Aircraft? 
B) Loss of Aircraft and Injury? 


C) Loss of Aircraft and Loss of Life? 
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SECTION IV 
INSTRUCTIONAL GUIDELINE DEVELOPMENT 
GENERAL 


The analysis of the selected decision tasks identified the characteristics 
of decision tasks involved in operational systems. Based on this information, 
a system for decision training guideline generation was designed to be used as 
a development aid for decision training programs in operational systems. The 
design objective was to develop a system to provide instructional guidelines 
presented in a form suitable for incorporation into the specification of 
instructional objectives, the planning of strategies, the selection of media, 
and the other steps required for the production of instructional materials for 
decision tasks. The system uses quantification schemes to incorporate expert 
knowledge into the guideline generation process. A systematic, step-by-step 
process identifies the system functions of the decision training guideline 
generator. The process involves (1) decision task identification, (2) decision 
type recognition, (3) decision task classification, (4) instructional content 
generation, (5) cognitive level weighting, (6) instructional method selection, 
(7) method/media relevance detection, and (8) instructional media selection. 

A description of the methodology, as well as the modules of the decision 
training guideline generation system, is the subject of the following: 


METHODOLOGY 


Current efforts in the development of decision training systems have 
usually evolved in the context of task-specific systems (e.g., medical, 
tactical, managerial, etc.). A training system for decision makers that is 
developed within the context of such a task-specific domain, without incor- 
poration of general purpose decision training components, does not provide 
trainees with the skills required for making task-specific decisions within 
other domains. Furthermore, since the characteristics of decision tasks 
within any specific domain may be subject to change with time, such training 
systems will not even provide useful long-term guidelines within the trainee's 
own domain. In these systems, the decision maker learns a tailor-made set of 
conceptual components, procedures, techniques, heuristic rules, parameters, 
and values unique to that particular system at the time of training (Figure 
11). The set of concepts that the potential decision maker learns is in 
reality a very specific subset of more general conceptual components, 
procedures, and techniques. Yet, he never becomes aware that, in fact, it is 
a subset of the more general set because it is outside the training boundary. 
Hence, if a decision maker is transferred to a new decision making domain, or 
the characteristics of the decision tasks within his decision domain change, 
he typically has two choices: he must either enter a new task-specific 
decision training program (and learn another unique conceptual subset again 
without realizing its relationship to the first subset learned), or he must 
learn to invent his own conceptual set without the benefit of a formal 
training program. In the latter case, the decision maker is without a formal 
guide, and often attempts to "cram" his new world into a conceptual subset 
learned for an entirely different decision task. 
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In the development of the decision training guideline generation 
system, transferability was one of the basic design considerations. Al- 
though the project has been initially aimed to develop a systematic pro- 
cedure for providing instructional guidelines for decision tasks encountered 
in LAMPS ASW Operation, a generalized methodology was developed to underlie 
the design in order to make the application of the system to other task 
domains subject to minimal modification effort. 


The project started with an analysis of tasks involved in the operator's 
job. Decision tasks were identified and analyzed. The result of the 
analysis was a detailed decision tree for each decision task which provided 
the task-specific knowledge. For each task, the parameters involved were 
identified and quantified. Heuristic rules and techniques relevant to 
decision making process for the tasks were selected and considered for the 
training program. However, a training system development approach based 
solely on the above task specific knowledge, without incorporation of 
general purpose decision training components, will limit the scope of the 
system. To avoid this problem and to provide for adaptability to new 
decision situations, information concerning general decision making be- 
havioral objectives were included in the design of the system. 


DECISION TRAINING BEHAVIORAL OBJECTIVES. Although a task specific decision 
training system is built around a nucleus of task specific knowledge, its 
primary objective is to create the required decision making "behaviors" in 
trainees. Since there is a common underlying set of skills required for 
performing every decision making task, a set of general decision making 
behaviors, associated with those skills, can be identified as behavioral 
objectives for any decision training system. These behavioral objectives 
will provide the required general purpose decision training components for 
the system. Identification of such decision training behavioral objec- 
tives is the subject of this section. 


Different structures of conceptualizing decision processes have been 
proposed. Some conceptualizations emphasize differences among decision 
situations; other focus on the tasks that decision makers are required 
to perform. Nickerson and Ferhrer (1975) present an overview of these 
structures. 


Adelson (1961) proposed a taxonomy of decision tasks that are carried 
out in modern command and control systems: 


a- Characterization of the state of the world 

b- Determination of the available action alternatives 
c+ Outcome prediction 

d- Choice rationalization 


Six steps for a business executive decision making process have been 
identified by Drucker (1967) as: 


a.» Classification of the problem 


b. Definition of the problem 
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c- Specifications that the answer to the problem must satisfy 
d- Decision as to what is right 
e. Building into the decision, the action to carry it out 


f. Feedback which tests the validity and effectiveness of the 
decision against actual course of events. 


The following thirteen steps are suggested by Edwards (1965) as different 
categories of activities that must be carried out by any Bayesian decision 
maker: 

a- Recognition of the existence of a decision problem 
b- Identification of available actions 


c. Identification of relevant states that determine payoff for — 
actions 


d- Identification of the value dimensions to be aggregated into the 
payoff matrix 


e- Judgmental evaluation of each outcome and each dimension 
f- Aggregation of value judgments into a composite payoff matrix 


g- Identification of information sources relevant to discrimination 
among states 


h- Data collected from information sources 

ij. Data filtration 

j. Estimation of likelihood ratios 

k- Aggregation of impact estimates 

1. Selection of best alternative by expected value maximization 
m- Implementation of the decision. 


Hil] and Martin (1971) have recognized six different steps involved in 
the decision making process. 


a- Identification of concern 
b- Diagnosis of situation 
c. Formulation of action 


d- Test of feasibility of selected alternatives 
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e. Adoption of alternative 
f. Assessment of consequences of adopted alternative. 


A more structured conceptualization of the decision making process 
has been performed by Schrenk (1969): 


a. Problem recognition, involving: 


Information acquisition 

Objective recognition 

Decision need perception 

Problem urgency and importance assessment 


b. Problem diagnosis, involving: 


Definition of possible situations 

Evaluation of situation likelihoods 

Determination of whether more information is needed 
Identification of possible data sources 

Judgment of value versus cost 

Seeking of information 

Diagnosis of decision situation 

Determination of whether alternatives under consideration 
account for all the data 

Diagnostic decision. 


c. Action Selection, involving: 


Definition of action goals 

Specification of value and time criteria 

Weighting of decision criteria 

Specification of risk philosophy 

Consideration of operating doctrine 

Generation of action alternatives 

Prediction of possible outcomes 

Estimation of outcome gains and losses 

Estimation of outcome likelihoods 

Evaluation of expected values of action versus their costs 
Evaluation of actions by risk philosophy 
Determination of whether more information is needed 
Seeking of information 

Re-evaluation of action alternatives 
Determination of whether the action if acceptable 
Selection of course of action 

Implementation of course of action. 


Soelberg (1966) suggests six aspects of the decision making process as: 
a- Problem recognition 


b- Problem definition 
c- Planning 
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d. Search 
e, Confirmation 
f. Implementation 


An analysis of these alternative conceptualizations reveals the common 
underlying structure existing in all of them. This common structure iden- 
tifies a decision task to be composed of either or both of two major sub- 
tasks: problem structuring and alternative selection. Each major subtask, 
in turn, may be further subdivided into a number of smaller subtasks. 
Problem structuring can be divided into alternatives formulation and out- 
come identification; alternative selection can be partitioned into utility 
assessment, probability assessment, decision rule application, and best 
alternative selection (Figure 12). The possible transition between sub- 
tasks in a decision making process is outlined in Figure 13. 


The six subtasks appearing as tip nodes in Figure 12 represent the 
general decision making components which must be included in a decision train- 
ing program. These components are restated as instructional objectives: 


a. Formulate alternative courses of action for a given decision 
situation. 


b. Establish outcomes of a given course of action for a given 
decision situation. 


c. Assess the utility of a given outcome in a given decision 
situation. 


d. Assess probabilities of a given set of events which may occur 
as a result of a given alternative course of action in a given 
decision situation. 


e. Apply a relevant decision rule to a given decision situation. 


f. Select the best alternative course of action for a given 
decision situation. 


Each instructional objective is defined in terms of specific learning 
outcomes, that is, a representative sample of specific types of behavior 
that is to be used as evidence that the objective has been achieved. The 
derived behavioral objectives associated with each instructional objective 
are shown in Table 14. 


INSTRUCTIONAL GUIDELINE GENERATION SYSTEM 


Based on the methodology developed in this chapter, a system was de- 
signed to provide guidelines for the development of training programs in 
the area of decision making. The guidelines include the suggestion of 
instructional contents, methods, and media compatible to the specific 
decision task as well as characteristics, requirements, and resources of 
the training center. 
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TABLE 14. DECISION TRAINING BEHAVIORAL OBJECTIVES 


Formulate alternative courses of action for a given 
decision situation. 


1.1 Identify the variable(s) and parameter(s) in 
the decision situation. 


1.2 Classify each of the variables/parameters of 
a given set as relevant or irrelevant to the 
decision situation. 


1.3 Identify constraints on each variable/parameter 
based on the statement of decision situation 
and general domain knowledge. 


1.4 Define solution range (set) of the variable(s) 
for the decision situation. 


1.5 Generate a set of alternatives using some 
or all of the variables/parameters. 


1.6 Identify those alternatives which are feasible 
in light of constraints on individual variables, 
statement of decision situation, and general 
domain knowledge. 
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TABLE 14. DECISION TRAINING BEHAVIORAL OBJECTIVES (CONTINUED ) 


2. Establish outcomes of a given course of action for a given 
decision situation. 


2.1 State the possible events that may occur as a result 
of application of the alternative course of action to 
the decision situation. 


2.2 Identify the variables in the decision situation 
that will be affected if a particular event occurs 
as a result of application of the alternative 
course of action to the decision situation. 


2.3 State the resulting decision situation in terms of 
the variables when a particular event occurs as a 
result of application of the alternative course of 
action to the decision situation. 
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TABLE 14. DECISION TRAINING BEHAVIORAL OBJECTIVES (CONTINUED) 


3. Assess the utility of a given outcome in a given decision 
situation. 


3.1 State the possible external effects of the outcome 
on the decision situation. 


3.2 State the likelihood that a given external effect 
will result from the given outcome in the decision 
situation. 


3.3 Identify whether a given effort of the outcome is 
favorable or unfavorable for the decision situation. 


3.4 State the degree of importance of a given external 
effect resulted from the outcome from the decision 
situation. 


3.5 Estimate the aggregate utility of the outcome based 
on the given possible external effects with given 
chances of occurrence, given direction of effect 
(negative effect or positive effect), and given 
magnitude of importance. 
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TABLE 14. DECISION TRAINING BEHAVIORAL OBJECTIVES (CONTINUED) 


4. Assess probabilities of a given set of events which 
probably will occur as a result of a given alternative 
course of action in a given decision situation. 


4.1 Identify the event with the highest chance of 
occurrence among the set of events. 


4.2 Sort the set of events according to the order of 
likelihood of occurrence in the decision situation. 


4.3 Assign likelihood values to the given ordered list 
of events in relation to the decision situation. 


4.4 Calculate probabilities of events for the given 


ordered list of events with given likelihood 
values by normalizing the likelihood values. 
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TABLE 14. DECISION TRAINING BEHAVIORAL OBJECTIVES (CONTINUED) 


Apply a relevant decision rule to a given decision 
situation. 


5.1 Name all decision rules. 


5.2 Describe the procedure for applying each 
decision rule. 


5.3 State the advantages and disadvantages involved 
in application of each decision rule. 


5.4 Define the criteria and procedure for selection 
of a decision rule based on the characteristics 
of the given decision situation and advantages/ 
disadvantages of the available decision rules. 


5.5 Select the best decision rule for the given 
decision problem by comparing each decision 
rule in terms of the criteria defined and 
selecting the most desirable one on the basis 
of relative merits with respect to the decision 
problem. / 


5.6 Apply a given decision rule to the given decision 
problem. 
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TABLE 14. DECISION TRAINING BEHAVIORAL OBJECTIVES (CONTINUED) 


Select the best alternative course of action for a 
given decision situation. 


6.1 Name all selection criteria for the best alternative. 


6.2 State selection criteria relevant to a given 
decision rule. 


6.3 Select the best selection criterion for the 
given decision situation, based on a given 
decision rule. 


6.4 Apply a given selection criterion to the given 


decision situation after the application of a 
given decision rule. 
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The instructional guideline generation system is basically composed 
of two modules: Decision Task Classification, and Decision Training Guide- 
line Generator (Figure 14). A task, which has been already identified as a 
decision task, enters the Decision Task Classification module. This 
module acts as a preprocessor which associates the task with one of the 
decision classes relevant to the training objectives. The output will be 
class information which enters the Decision Training Guideline Generator 
Module. Based on the classification result, as well as training environ- 
ment characteristics, requirements, and resources, the module suggests 
the. required instructional content and the best instructional methods and 
media for the input decision task. 


A detailed diagram describing the relationships among different com- 
ponents of the system is shown in Figure 15. A brief description of the 
eight components and their interactions as well as the input/output of the 
system is described in the remainder of this section. In what follows, 
attention must be paid to the fact that the implicit procedure defined by 
the system is being performed (operated) by an instructional technologist. 


Decision Task Identification receives any training task presented in 
the objective hierarchies and works as a filter which separates decision 
tasks from non-decision tasks. The specification of this component follows 
the definition of a decision task: 


a. The objective of a decision task is to select an alternative 
from a specified set of alternatives. 


b. This selection may require the formulation of alternatives 
(problem structuring). 


c. There is a lack of completely specified criteria for either 
alternative formulation or alternative selection. 


The output of the component is the identification of the input task 
either as a decision task, which causes the task to enter the Decision Type 
Recognition and Decision Task Classification components, or as a-non- 
decision task, which results the initiation of a conventional ISD procedure. 


Decision Type Recognition receives a decision task and identifies it 
as Type 1, Type 2, or Type 3. Type 1 represents decision tasks with 
processes requiring only problem structuring; Type 2 includes all decision 
tasks concerning only action selection; Type 3 is composed of all decision 
tasks which require both problem structuring and action selection. The 
information concerning the type of the decision task enters Decision Task 
Classification and Cognitive Level Weighting components. 


Decision Task Classification is based on a procedure similar to that 
of discriminant analysis. To distinguish between classes of decision tasks, 
a collection of discriminating variables is selected that measures charac- 
teristics on which the classes are expected to differ, that is, training 
content requirements. Thirty different classes constitute the set of 
possible classes relevant to decision training. The decision task and 
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the information concerning its type enter the Decision Task Classification 
component. The output of the component represents the class of the deci- 
sion task which is sent to the Instructional Content Generation component. 


Instructional Content Generation receives the information about the 
Class of the decision task and generates the required instructional contents. 
Since a set of basic instructional contents is associated with each class, 
the process involved is a simple table look-up. The basic instructional 
content appears as part of the system's output which together with the task- 
specific information presented in decision trees, constitute required 
instructional content for the decision training program. 


Cognitive Level Weighting receives the decision type and selects a 


cognitive level weight vector. The idea of a cognitive level weight vector 
was initiated by the fact that the training objectives for different tasks 
with unequal cognitive complexity can be accomplished differently depending 
on the instructional method employed. Such dependency between cognitive 
complexity and instructional method suggested a detailed analysis of cog- 
nitive levels involved in the performance of each decision behavior. The 
result of the analysis was the identification of a cognitive level weight 
vector associated with each decision type. Each vector presents the degree 
of importance of the six cognitive levels with respect to the corresponding 
decision type. The six cognitive levels include: knowledge, comprehension, 
application, analysis, synthesis, and evaluation. Since a weight vector is 
assigned to each decision type, the process involved is a simple table 
look-up. The selected cognitive level weight vector is sent to Instruc- 
tional Method Selection component. 


Instructional Method Selection receives the cognitive level weight 
vector, incorporates it with the data in the method/attribute matrix, and 
results in the most effective instructional method. The method/attribute 
matrix represents the judgmental evaluation of the degree of effectiveness 
of fourteen major instructional methods with respect to the six cognitive 
levels. The judgmental data was provided by a training specialist and a 
psychologist. A weighted average procedure is used to calculate a degree 
of effectiveness for each instructional method with respect to the deci- 
sion task. The method with the highest degree of effectiveness is selected 
as the most effective instructional method and appears as part of the 
system's output. 


Method/Media Relevancy Detection receives the selected instructional 
method and, using the method/media relevancy matrix, identifies the media 
relevant to the selected method. Since some instructional media are not 
applicable to a training environment employing a specific instructional 
method, a study of the relevancy of each of the twelve instructional media 
with respect to each instructional method was performed. The result of the 
study dichotomized the relevant and irrelevant media for each instructional 
method. These results appear in the method/media relevancy matrix. The 
relevant instructional media are identified by a reference to the method/ 
media relevancy matrix, and the information is sent to the Instructional 
Media Selection component. 
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Instructional Media Selection receives the relevant media and, using 
the media/attribute matrix and the weights of different attributes with respect 
to the training environment, selects the most effective media. The degree 
of effectiveness of each instructional media with respect to different 
attributes was judgmentally assessed by a training specialist. During 
system's application, the system user instructional technologist will provide 
a weight vector which identified. the degree of importance of different 
attributes of the instructional method with respect to the specific. training 
environment. Then, a weighted average procedure will be used to calculate 
the degree of effectiveness of each relevant instructional media with respect 
to the decision task. The media with the highest degree of effectiveness is 
selected as the most effective instructional media and appears as part of 
the system's output. 


DECISION TASK CLASSIFICATION PREPROCESSOR 


The decision task space can be divided into different classes such that 
each class represents a set of decision tasks with some degree of similarity 
with respect to their training requirements. The inter-class similarity 
will then suggest implementation of similar instructional content, method, 
and media for all decision tasks belonging to the same class. The classifica- 
tion scheme must be oriented toward training and its discriminant elements 
must be designed based on decision task differences which suggest different 
training strategies. 


Such a classification scheme was designed with two components: decision 
type recognition and decision task classification. Decision type recognition 
is based on the methodology described in Chapter 2. Three different types of 
decision tasks are identified. Type 1 represents decision tasks with 
processes requiring only problem structuring; type 2 includes all decision 
tasks concerning only action selection; type 3 is composed of all decision 
tasks which require both problem structuring and action selection. Since 
the processes involved in problem structuring are of different cognitive 
complexity than the ones involved in action selection, the training processes 
for the tasks belonging to different decision types must be conducted differ- 
ently. Such differences in training processes will be specified in terms of 
the guidelines for instructional content, method, and media. 


In constructing the second component, decision task classification, the 
objective is to provide a procedure which assigns each decision task to a 
category associated with the most relevant instructional strategy. If all 
decision tasks can be so categorized into a given number of classes, and if 
training can be developed for each class, then any decision task may be 
easily categorized and appropriate training programs easily developed for it. 


This procedure is similar to that of discriminate analysis. Discrimi- 
nant analysis begins with the desire to statistically distinguish between 
two or more groups of cases (in this instance, a case would be decision task 
and discriminant analysis would be a means of distinguishing among two or 
more classes of decision tasks). To distinguish between classes, a collection 


of discriminating variables is selected that measures characteristics on which 
the classes are expected to differ. Ideally, discriminating variables are 


78 


NAVTRAEQUIPCEN 77-C-0005-1 


assigned to classes in such a way that the separation of the classes is 
maximized. A set of variables was designed which provides satisfactory 
discrimination for decision tasks with known class membership. The classes 
have been identified, and the classification of any new decision task with 
unknown membership can be accomplished. 


The key to developing a decision task classification scheme is the 
identification of the variables or decision task attributes to be used as 
discriminating variables. 


There are many attributes associated with decision making tasks. The 
following represents a preliminary listing of the ones most relevant to a 
training program. 

a- Single attribute - multi-attribute 

b- Individual - group 

c+ Static - dynamic 

d+ Qne shot - repetitive 

e- Certainty - risk/uncertainty 

f. Abstract (general) - concrete (task specific) 


g: Well defined - ambiguous 


h. Decision making - decision execution (Note: this characteristic 
follows a convention developed in this project.) 


i- Time critical - time relaxed 
j- Small probability high loss - normal ranges for probability and loss 


k- Type 1 (problem structuring), type 2 (alternative selection), 
and type 3 (problem structuring and an alternative selection). 
(Note: This characteristic follows a convention developed in 
this project.) 


1. Decision rule (SEU, EV, Mini-max, Maxi-min, etc.) 


Each attribute describes a different dimension of decision making and represents 
a different "pigeon hole" in which a decision task could be categorized. Given 
this set of 12 attributes, and allowing for seven different and mutually 
exclusive types of decision rules provides a very modest listing of decision 
classes, compared to all possible classes that are potentially available to 
decision makers. However, even with such a list, the implications as to the 
number of classes into which decision tasks could be discriminated is 

enormous. Clearly, this many classes is neither feasible nor desirable. What 
assumptions can be reasonably made to reduce this number of classes to a more 
manageable number? First, since we are interested in training ATO's and 
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Pilots in the LAMPS ASW operation, we can assume that the decision tasks, 
and hence training, will focus on individual decision makers. Next, assume 
that all decisions are repetitive decisions. That is, although a decision 
is unique to a particular ASW mission and the parameters of that mission, an 
outside observer would find that there is a great deal of similarity in the 
decision tasks across all pilots and ATO's and across all ASW missions. For 
example, the decision of what sonobuoy pattern to drop occurs in each and 
every ASW mission. Also, assume that each decision is task specific rather 
than general. Each decision must contribute to overall mission effectiveness 
and, hence, must be task specific at the point in the mission at which it is 
made. Moreover, assume that since we are interested in training decision 
makers, we are not concerned here with decision executions that are the 
result of previously made decisions. It is also valid to assume that all 
decisions are made under risk and in normal ranges of probability and loss. 
At this point it is useful to refer to Figure 16 and Table 15 which is a more 
detailed representation of what a decision task classification preprocessor 
miaht look like. Note that a decision task follows one of three major flows: 
(1) type 1 decision, (2) type 2 decision, or (3) type 3 decision. Types | 
and 2 are of major interest, since type 3 is merely a combination of types 

1 and 2. Type 1 focuses on problem structuring. The attributes of interest 
here are: 


a- Single attribute - multi-attribute 
b. Well defined - ambiguous 

é: Time relaxed - time pressure 

d. Static - dynamic 


Since type 1 is only limited to problem structuring, decision rules are not 
relevant. The focus is on alternative generation and outcome estimation. 
Therefore, for the type 1 flow, there are 2 x 2 x 2 x 2 = 16 possible classes 
into which decision tasks may fall. For type 2 decision tasks, those that 
focus on alternative selection, the following attributes are relevant: 


a. Time relaxed - time pressure 
b. Decision rules 


Subjective expected utility (SEU) 
Maxi-min utility (MMN) 

Maxi-max utility (MMX) 

Mini-max regret (MMR) 
Lexicography (LEX) 

Hurwicz (HUR) 

Satisfying (SAT) 


Since the focus here is on alternative selection, there are 2 x 7 = 14 classes 
into which type 2 decision tasks may be sorted. Type 3 decision tasks can 

be sorted using the type 1 and type 2 classes in conjunction. There are, 
therefore, 16 x 24 = 224 classes into which type three decision tasks may 

be sorted, but since these classes are simply combinations of type 1 and type 2 
tasks they do not need to be separately identified. When classifying decision 
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Figure 16. Decision Task Classification 
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TABLE 15. DECISION TASK CLASSES 


Single attribute, 
Single attribute, 
Single attribute, 
Single attribute, 
Single attribute, 
Single attribute, 
Single attribute, 


Single attribute, 


ATTRIBUTE 


well defined, time relaxed, static 
well defined, time relaxed, dynamic 
well defined, time pressure, static 
well defined, time pressure, dynamic 
ambiguous, relaxed, static 
ambiguous, relaxed, dynamic 
ambiguous, pressure, static 


ambiguous, pressure, dynamic 


Multiattribute, well defined, time relaxed, static 


Multiattribute, well defined, time relaxed, dynamic 


Multiattribute, well defined, time pressure, static 


Multiattribute, well defined, time pressure, dynamic 


Multiattribute, ambiguous, time relaxed, static 


Multiattribute, ambiguous, time relaxed, dynamic 


Multiattribute, ambiguous, time pressure, static 


Multiattribute, ambiguous, time pressure,- dynamic 
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TABLE 15. DECISION TASK CLASSES (CONTINUED) 


I 


DECISION 
CLASS 
NUMBER TYPE CLASS ATTRIBUTE 
7 2  RSEU Time relaxed, subjective expected utility 
18 2  PSUE Time pressure, subjective expected utility 
19 2 RMMN Time relaxed, Maxi-min utility 
20 2 PMMN Time pressure, Maxi-min utility 
21 2 RMMS-~ Time relaxed, Maxi-max utility 
22 2  PMMX Time pressure, Maxi-max utility 
23 2  RMMR- Time relaxed, Mini-max regret 
24 2 PMMR Time pressure, Mini-max regret 
25 2  RLEX Time relaxed, Lexicography 
26 2 PLEX Time pressure, Lexicography 
27 2  RHUR- Time relaxed, Hurwicz 
28 2  PHUR Time pressure, Hurwicz 
29 2 RSAT Time relaxed, satisfying 
30 2  PSAT Time pressure, satisfying 
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tasks in this manner, 30 different classes are required provided that the 
decision tasks are initially presorted into type 1, 2, or 3 and that type 3 
decision tasks can be trained by taking the relevant type 1 and type 2 tasks 
in conjunction. Thirty decision classes is a reasonable and manageable number 
of classes for which training materials can be developed and tested. The 
resulting decision classes are shown in Table 15. 


Type 3 decision tasks that enter the system are diagnosed serially through 
the type 1 and type 2 decision classes. For example, a decision task that is 
type 3 could appear as 3-MMPS-PMMR which is (1) multiattribute, well defined, 
time pressure, static, for alternative generation and outcome estimation, and 
(2) time pressure, mini-max regret for alternative selection. Note that some 
combinations cannot exist (or are unlikely to exist) such as time pressure in 
type 1 and time relaxed in type 2. To train for type 3 - MWPS-PMMR, the system 
will simply take the training package for 1-MWPS and the training package for 
2-PMMR and form one complete training package. The structure of the decision 
task space for training, resulting from the classification scheme, is shown in 
Figure 17. 


INSTRUCTIONAL CONTENT GENERATOR 


Identification of different implications of each decision class for 
training purposes is a task which must follow the development of the 
classification scheme. Such implications must reveal the basic content 
requirements for each class. Identification of these implications is the 
subject of the following. The training implications of each class are 
represented in terms of a set of modules corresponding to possible attributes 
of a decision task defined for the task classification scheme. 


The basic instructional content of each class will be a combination of 
the contents associated with the attributes of that class. However, in a task- 
specific decision training context, such as the present program, the 
instructional content for each class will be affected by the task-specific 
information which serves as a vehicle for conveying the basic content. Such 
task-specific information is provided by decision trees constructed for each 
decision situation. For example, if utility is a basic training content and 
the task is sonobuoy patterns, then the utility of sonobuoy patterns must be 
included in the instructional content. 


During this program, the implications of the relevant decision task 
attributes and decision rules were analyzed. The analysis emphasized the 
training aspects and included the selected eight attributes (Single Attribute, 
Multiattribute, Well Defined, Ambiguous, Time Relaxed, Time Pressured, Static, 
and Dynamic) and the seven decision rules (Subjective Expected Utility, Maxi- 
min Utility, Maxi-max Utility, Mini-max Regret, Lexicography, Hurwicz, and 
Satisfying). The result was the identification of the basic instructional 
content required for each decision attribute. This information was structured 
in fifteen modules, each module represents information concerning General 
Description, Amplification, Rules, Pitfalls/Limitations, Interacting with 
Other Decision Elements, and Prerequisites associated with the corresponding 
attribute or decision rule. Due to the large volume of the information, and to 
avoid the disparity of thought, the results are presented in Appendix J. 
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COGNITIVE LEVEL WEIGHTING 


The training objectives for different tasks with unequal cognitive 
complexity can be accomplished differently depending on the instructional 
methods employed. Such dependency between cognitive complexity and instruc- 
tional method suggests a detailed analysis of cognitive levels involved in 
performance of each decision behavior required for different decision making 
components. 


Such an analysis was performed to identify the highest cognitive level 
involved in the decision making behaviors. Six cognitive levels are con- 
sidered as the major levels relevant to the process of decision training: 
knowledge, comprehension, application, analysis, synthesis, and evaluation. 
These levels are identical to the major categories in the cognitive domain 
of the taxonomy of educational objectives described in Bloom, 1956. We 
chose to call them cognitive levels instead of cognitive categories to 
establish, implicitly, a mental picture of the hierarchical cognitive 
complexity existing among different cognitive levels. The description of 
the six cognitive categories have been summarized in Gronlund, 1970: 


"Knowledge ts defined as the remembering of previously learned 
matertal. Thts may involve the recall of a wide range of 
matertal, from spectfie facts to complete theortes, but all 
that ts requtred ts the bringing to mind of the appropriate 
information. Knowledge represents the lowest level of 
learning outcomes tn the cognitive domatns. / 


"Comprehenston is defined as the abtlity to grasp the meantng 
of matertal. Thts may be shown by translating matertal from 

one form to another (words to numbers), by interpreting matertal 
(explaining or summartzing), and by estimating future trends 
(predicting consequences or effects). These learning outcomes 
go one step beyond the stmple remembering of material, and 
represent the lowest level of understanding. 


"Application refers to the abtlity to use learned material in 

new and conerete situations. Thts may tnelude the applicatton 
of such things as rules, methods, concepts, princtples, laws, 

and theories. Learning outcomes in thts area require a higher 
level of understanding than those under comprehension. 


"Analysts refers to the abtltty to break down material tnto 
tts component parts so that tts organizational structure may 
be understood. This may itnelude the identification of the 
parts, analysts of the relattonshtps between parts, and 
recognition of the organizational principles tnvolved. 
Learning outcomes here represent a higher intellectual level 
than comprehension and application because they require an 
understanding of both the content and the structural form 
of the matertal. 
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"Syunthests refers to the ability to put parts together to form 
anew whole. Thts may tnvolve the production of a untque 
communteatton (theme or speech), a plan of operations (research 
proposal), or a set of abstract relations (scheme for classifying 
tnformatton). Learning outcomes in this area stress creative 
behaviors, with major emphasis on the formulation of new patterns 
or structures. 


"Evaluation ts concerned with the abiltty to judge the value 

of matertal (statement, novel, poem, research report) for a given 
purpose. The judgments are to be based on definite criteria. 
These may be tnternal eriteria (organtzation) or external 
ertterita (relevance to the purpose) and the student may determine 
the eriterta or be given them. Learning outcomes tn thts area 
are highest in the cognitive hterarchy because they contain 
elements of all of the other categortes, plus consctous value 
judgments based on clearly defined ertteria." 


The result of the analysis identified the highest cognitive level 
involved in each behavior as well as the degree of importance of that cognitive 
level for creating the corresponding behavior in a decision making context. 
The analysis was based on the decision training behavioral objectives developed 
previously (Table 14). The result of this analysis is summarized in Table 16. 
The column with entries identifies the highest cognitive level associated 
with the corresponding behavior. The entry itself represents the degree of 
importance of the corresponding cognitive level/decision behavior pair. 


In order to assign the entries, the importance of each of the subtasks 
involved in a decision making process was analyzed and three different levels 
of importance were identifed. An entry equal to 1/2, 1 or 2 was assigned to 
the subtasks in low, middle or high level of importance, respectively. Such 
assignment of entries resulted in equal values for the summation of the weights 
assigned to each of the two major subtasks. Since problem structuring and 
alternative selection are of similar importance to a decision making process, 
resulting equal values can be considered as a confirming evidence to the 
validity of this method of assignment. 


Since Type 1 decision tasks involve only alternative formulation and 
outcome estimation, the summation of all degrees of importance associated 
with each cognitive level corresponding to behaviors relevant to these two 
decision subtasks provides an element of a weight vector for cognitive levels 
associated with Type 1 decision tasks. Such a summation was performed for 
all cognitive levels to provide the complete cognitive level weight vector 
for Type 1 decision tasks. A similar procedure was performed on behaviors 
relevant to utility assessment, probability assessment, decision rule applica- 
tion, and best alternative selection subtasks to provide cognitive level 
weight vector for Type 2 decision tasks. Since Type 3 decision tasks involve 
_ the subtasks relevant to both Type 1 and Type 2 decisions, the cognitive level 
weight vector for Type 3 decision tasks was resulted by simply adding the 
corresponding weights obtained for Type 1 and Type 2 decision tasks. The 
weight vectors for the three decision types are shown in Table 17. The elements 
of each vector have been normalized such that the summation of elements adds up 
to one. 
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TABLE 16. HIGHEST COGNITIVE LEVEL IDENTIFICATION AND WEIGHT 


ASSIGNMENT FOR DECISION MAKING BEHAVIORS 


Cognitive 
Level 


Knowledge 
Comprehension . 
Application 


Formulate Alternatives 


Identify variables 2 
Classify variables , 
Identify constraints 

Define solution range 

Generate alternatives 

Identify alternatives 


a ee ee ee 
° . . . . . 
nn FF wn 


Establish Outcomes / 


2.1 State events 2 
2.2 Identify variables 
2.3 State resulting situations 2 


Assess Utilities 


3.1 State effects 

3.2 State likelihoods 

3.3 Identify favorability 1 
3.4 State importance | 1 
3.5 Estimate utilities ] 


88 


Analysis 


Synthesis 


Evaluation 


NAVTRAEQUIPCEN 77-C-0005~-1 


TABLE 16. HIGHEST COGNITIVE LEVEL IDENTIFICATION 
AND WEIGHT ASSIGNMENT FOR DECISION 
MAKING BEHAVIORS (CONTINUED) 


Cognitive 
Level 


Decision 
Behavior 


Knowledge 
Comprehension 
Application 
Analysis 


4, Assess Probability 


4.1 Identify event ] 
4.2 Sort events ] 

4.3 Assign likelihoods 

4.4 Calculate probabilities ] 


5. Apply Decision Rule 


5.1 Name rules 


ww 


5.2 Describe procedure ] 
5.3 State advantages 1] 
5.4 Define criteria 

5.5 Select best rule ] 
5.6 Apply rule 2 


6. Select Best Alternative 


es 


6.1 Name criteria 


we 


6.2 State relevant criteria 
6.3 Select criteria 


wv 


6.4 Apply criteria 


ne 
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A high value element represents a high degree of importance of the 
corresponding behavior with respect to the decision type. Lower value 
elements define a lesser degree of importance for the corresponding behaviors. 
The degrees of importance of the lowest cognitive level lknawiedge) and the 
highest cognitive level (evaluation) for Type 1 decision tasks are very low. 
These degrees can be approximated by zero without resulting substantial 
inaccuracy. The cognitive level weights for the three decision types are 
plotted in Figures 18, 19, and 20. These figures indicate that the resulted 
weights correspond to intuitive judgment. The occurrence of the maximum 
degree of importance for Type 1 around Analysis level is justified by the 
fact that the process of problem structuring mainly involves analysis of the 
problem in order to identify the relevant eteneats of the problem as wel] 
as the anlaysis of possible outcomes in order to identify their effects on 
the problem elements. Since Type 2 decision tasks mostly involve application 
of certain rules and procedures, the maximum degree of importance for Type 2 
decisions occurs around Application level. For Type 3 decision tasks the 
degrees of importance are spread along different cognitive levels with 
relatively lower variances. In general the process of decision making 
mainly concentrates around Analysis (of the problem and potential solution 
outcomes) and Application (of a set of rules and procedures). 


INSTRUCTIONAL METHOD SELECTION 


The approach to solving the problem of instructional method selection was 
to establish a quantification scheme which can be used to rank different 
instructional methods as to their effectiveness for the training of any 
decision task. To implement this approach, an analysis of different instruc- 
tional methods was performed. Fourteen major instructional methods were 
selected and attributes of each method were identified. The methods include: 


a) Drill/Practice/Review 


b) Lecture 


( 

( 

(c) Discussion - Individual Tutorial 
(d) Discussion - Group 

(e) Programmed Instruction 
(f) Games 

(g) Simulation 

(h) Projects - Individual 
(i) Projects - Group 

(j) Laboratory 

(k) Apprenticeship 

(1) Demonstration | 
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Figure 18. The Degrees Of Importance Of Different 
Cognitive Levels For Type 1 Decision Tasks 
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Figure 19. The Degrees Of Importance Of Different 
Cognitive Levels For Type 2 Decision Tasks 
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Figure 20. The Degrees Of Importance Of Different 
Cognitive Levels For Type 3 Decision Tasks 
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(m) Individual Investigation (Research) 
(n) Text - Reading 


Based on the characteristics of each instructional method, the degree of 
effectiveness of that method for training different tasks with different 
levels of cognitive complexity were judgmentally evaluated. The judgmental 
criteria were of two fold: (1) the effectiveness of the method judged by 
the expected result of applying the method in a prototype training environ- 
ment and (2) the effectiveness of the method judged by implicit consideration 
of cognitive models of human learning. The judgmental data related to the 
first criterion was provided by a training specialist, and the data concerning 
the second criterion was provided by a psychologist. Lack of substantial 
variance between the corresponding judgmental values provided by the two 
criteria suggests itself as confirming evidence for the correctness of the 
judgments. However, an accurate assessment of these judgmental data demands 
a more refined procedure, performed in a controlled environment. Due to the 
importance of such data for research projects in the area of training 
cognitive processes, further research to elicit a more accurate assessment 
of the data appears to be needed. 


The judgmental data resulting from the two criteria were averaged to 
provide an aggregate assessment. The expected cost and time of instruction 
for each method were also assessed by the training expert. The results of 
the assessment is shown in Table 18 as Method/Attribute Matrix. The entries 
of the matrix have value ranges from one to ten. The higher the value, the 
more effective the method for training a task with the corresponding level 
of cognitive complexity. 


This information suggests a procedure for its utilization in instructional 
method selection. Once the type of the decision task to be taught becomes 
known, the cognitive level weighting vector relevant for the task is easily 
derived from the cognitive level weight matrix. Then, a linear evaluation 
procedure can be performed by a summation over the results of multiplication 
of the elements of the weight vector by corresponding elements of each vector 
associated with different methods. That is, if the cognitive level weight 
vector for the decision type associated with the task is defined as 
(Wy, s+. Wey «+. We),(the weights for "cost" and "time" with respect to the 
specific training Snvironment are assigned by the system user instructional 
technologist during system application as (wos We), and the vector associated 
with the .th method is defined as (m.,, .--5Ms.3 -++» Msg), then the 
effectivetiess value of the .th instrdetional method withfespect to the 
decision task will be: 


V5 = : Meg W. 


This procedure will provide an effective value assessment for each instruc- 
tional method. Then the method with the highest effectiveness value will be 
selected as the most effective instructional method for training the decision 
tasks under consideration. 
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TABLE 18. METHOD/ATTRIBUTE MATRIX 


ATTRIBUTES 


Comprehension 
Analysis 
Synthesis 
Evaluation 


Knowledge 
Application 


METHOD 


Drill/Practice/Review 


o ©] Cost (Low) 


Lecture 


Discussion-Individual Tutorial i 10 9 10 : 8 2 6 
Discuss ion-Group 4 9 7 8 7 5 3 
Programmed Instruction 8 8 5 4 i 5 8 
Games 3 3 8 7 5 4 4 3 
Simulation 2 3 10 8 8 7 2 3 
Projects-Individual 4 8 8 9 9 7 5 2 
Projects-Group 3 8 7 8 6 6 5 4 
Laboratory 4 6 10 9 5 8 2 4 
Apprenticeship 5 6 3 6 5 6 2 Z 
Demons traticr 

Visual Aid/Film 5 7 2 3 3 ] 5 6 
Individual Investigation 6 8 8 9 9 9 5 ] 
Text-Readings 8 7 3 4 3 2 10 7 
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If all the values involved in the 
instructional method were invariant, a 
instructional method for each decision 
type of a decision task would directly 
for the decision task. However, there 
on the characteristics of the training 


procedure for selection of the best 
method could be selected as the best 
type. Then, identification of the 
suggest the best instructional method 
are some elements whose values depend 
environment in which the training 


program will be implemented ("Cost" and "Time", in this case). The informa- 
tion concerning the characteristics of the training environment will be 
conveyed by the system user instructional technologist during system's 
application. This information will be presented in the form of judgmental 
importance weights assigned to a set of attributes. The set of attributes 
include: Learner paced control on the instructional media, instructional 
media to be self instructional, motion of the instructional media, fidelity 
of the instructional media in terms of situational details, learner control 
over the instructional speed, flexibility of the media for instructional 
sequencing, instruction readability, context creation, convenience for 
development of testing and evaluation, low cost of instruction, and simplicity 
of use and maintenance. Furthermore, introduction of the selection process 
as a dynamic procedure will provide for the possibility of modification of 
the judgmental values as well as addition or deletion of attributes. 
METHOD/MEDIA RELEVANCY 


Analysis of different instructional media identified twelve media of 
major interest: 


a. Oral Representation (Lecture) 

b. Textbook 

_C. Workbook/Problem Sets 

d. Pictures, Slides, Transparencies 
e, Audio Tape 

f. Motion Picture 

g. IV/VTR 

h. Programmed Text 

i. Computer Assisted Instruction (CAI) 
j. Part-Task Trainer 

k. Simulator 

1. Natural Object 


Since some of these instructional media are not applicable to a training 
environment employing a specific instructional method, a study of relevancy 
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of each instructional media to each of the selected instructional methods 

was performed. The result of the study dichotomized the relevant and irrele- 
vant media for each instructional method. These results appear in the 
Method/Media Relevancy Matrix, Table 19. The method/media relevancy matrix 
performs as a masking scheme. That is, each zero entry rules out the 
acceptability of an instructional media for the corresponding instructional 
method. A more detailed analysis can be performed to evaluate each method/ 
media pair judgmentally. The result of such analysis will be a degree of 
relevancy for each method/media pair. In this case the process of dichotomi- 
zation of media for each method will be replaced by a multilevel evaluation 
process which will provide more information. For example, although oral 
representation and textbook are both considered equally relevant to drill/ 
practice/review by the dichotomization of media, a multilevel evaluation 
might also convey the information that text book would be more relevant to 
drill/practice/review by assigning numbers on a scale of 1 to 10 as the 
degree of relevancy for the two media with respect to the method. 


INSTRUCTIONAL MEDIA SELECTION 

Once the best instructional method for a decision task has been 
selected, the relevant instructional media for the selected method can be 
easily identified by reference to the method/media relevancy matrix. The 
information concerning relevant media together with the information about 
the characteristics of the instructional environment provide the required 
data for selection of the best media. 

To design an instructional media selection methodology based on this 
information, an analysis of the characteristics, requirements, and 
resources of possible instructional environments was performed. The 
result of the analysis identified eleven attributes characterizing different 
instructional environments: 

a. Learner Paced (control) 

b. Self Instructional 

c. Motion/Still 

d. Situational Detail (Fidelity) 
e. Control Over Time (Motion) 

f. Flexible Sequencing 

g- Repeatability 

h. Context Creation 

i. Potential for Test/Evaluation 
j- Cost 


k. Simplicity of Use and Maintenance 
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TABLE 19. METHOD/MEDIA RELEVANCY MATRIX 


wu » 

o o x » 
pa x Cy es) 
i= un oO 3 - @ 
is) - e Pp : => 
- oe Vv a) 8 
pa n ee o ow - @® [= oO 

oO @ 4 x n 7 a a e °o 
Ps ° of UY 5 ic] - Ss » — 
cs ° oo £YNa » < o a) © o 
or a ar Ss uNW ° °o S to = & 
METHOD —- uu P xo PuUE 4 4 Sy oe = 5 
oo wv x L£o Orr © no) » fo} — & oc & + 
£& So o OS wee & 3 i=} x <= © S ed o 
oa~ e =a ante <x = a oO ae %,) =z 


Drill/Practice/ 

Review 1 1 1 1 1 } 1 1 1 1 ] } 
Lecture 1 0 0 0 1 ] 1 0 0 0 0 0 
Discussion- 

Individual Tutorial 1 0 0 0 0 0) 0 1 1 0 0 0 
Discussion-Group ] 0 0 0 0 0 0 0 0 0 0 0 
Programmed 

Instruction 0 0 1 1 1 1 1 a 1 }t 1 0. 
Games fy) 1 1 1 1 1 1 I 1 1 1 
Simulation 0 0 0 ] 1 1 1 0 } 1 1 1 
Projects- 

Individual 0 0. 0 0 0 0 0 0 0 0 0 1 
Projects-Group 0 0 0 0 0 0 0 0 6 0 0 } 
Laboratory 0 0 1 ] 1 1 1 } T 1 1 1 
Apprenticeship o 0 0 0 0 0 0 0 0 0 0 1 
Demonstration 0 0 0 1 0 ] } 0 0 1 T 1 
Individual 

Investigation 

(Research) 0 0 0 0 0 0 0 0 0 0 0 1 
Text-Reading 0 1 0 1 0 0 0 0 0 0 0 0 
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The degree of effectiveness of each instructional media with respect to 
different attributes was judgmentally assessed by a training specialist. The 
criterion for assessment in this case was the effectiveness of the media 
judged by the expected result of applying the media in a prototype training 
environment characterized by the corresponding attribute. The results of 
this assessment appears in Table 20. The entries of the matrix have value 
ranges from one to ten. The higher the value, the more effective the media 
for a training environment characterized by the corresponding attribute. 
Although the assessment process was made as refined as possible within the 
time limit of this project, a more detailed research of this subject, for 
eliciting more accurate assessment of the degrees of effectiveness, seems 
advisable. 


The Media/Attribute Matrix is used to select the best instructional media 
for a selected instructional method to be used in a specific instructional 
environment. The degrees of importance of attributes with respect to the 
training environment is defined as a vector presented by training designer. 
This vector acts as a weight vector which can be defined as (C,, ...; C.; 

...5 C,,). If the vector associated with the ,th media in media/attribute 
matrix!|s defined as (apps «--s-4kis sees ae and the vector aes with 
the selected ene in Method/Media relevancy matrix is defined as 

Ss ; then the degree of effectiveness for the , th media ak respect 
th? the ayee method and the instructional environment will be calculated as: 


E. = a j C. a 


The expression Ci a,; is a simple weighted average which results a number 
to represent an averalt effectiveness of the media with respect to the 
instructional environment. Depending on whether the media is compatible 

to the selected instructional method or not the value of rp may _be one or 
zero; this means that the value of E, will be equal to eethen ¥ Cy aq or. 
zero, respectively. Therefore, the result of applying this procedure to all 
twelve media will be twelve numbers each representing the overall effective- 
ness of a media based on both the characteristics of instructional environ- 
ment and the selected instructional method. The method with the highest 
overall effectiveness value will be selected as the most effective media to 
be used with the selected instructional method in the specific instruction 
environment. 


PROCEDURE FOR INSTRUCTIONAL GUIDELINE GENERATION 


The process of instructional guideline generation for decision tasks is 
compatible with conventional Instructional System Development (ISD). This 
process can be considered as a module, within a structured ISD program, for 
recognition and classification of decision tasks as well as selection of the 
most effective instructional methods, media, and contents for those tasks. 
The process of instructional guideline generation can be defined as a syste- 
matic execution of a series of steps: 


a. Define Objective Hierarchies for the Training Tasks. As in the 
~ case of any ISD program, the process of instructional guideline 


generation starts with identification of objective hierarchies 
for the training tasks. 
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TABLE 20. 


ATTRIBUTES 


Learner 
Paced 
(Control) 


MEDIA 


Lecture 


(oral presentation/chalk 
board) 1 
Textbook 10 
Workbook, Problem Sets 10 
Pictures, Slides, 
Transparencies 10 
Audiotapes 5 
Motion Pictures 3 
Television/Videotape 
Recording 3 
Programmed Text 10 
CAI 10 
Part-task Trainer 10 
Simulator 10 
Natural Objects 3 


Instructional 


Self 


MEDIA/ATTRIBUTE MATRIX 


Motion/Skil] 


Situational 


Detail 


(fidelity) 


10] 


Control over 


Flexible 
Sequencing 


10 


10 


Repeatable 


Context 
Creation 


Allows 


Testing/ 


Evaluation 


10 
10 


Simplicity 
of Use, 


10 
10 


Maintenance 
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Recognize Decision Tasks. The objective hierarchies are processed 
by the decision task identification scheme to recognize the decision 
tasks involved in the training tasks. 


Identify Type and Class of the Decision Task. Using the method 
described as decision task classification preprocessor, the type 
and the class of each decision task are identified. 


Select the Instructional Contents. The instructional content sections 


corresponding to different attributes of the class of decision task 
are combined to construct the basic instructional contents. The 
basic instruction content includes: general definition, amplifi- 
cation, rule, pitfalls and limitation, interactions with other 
decision elements, and prerequisites for each instructional content 
section. In addition to basic instructional content, the decision 
tree, representing each decision task, is constructed to provide 
the required task-specific information for instructional content. 


Identify Cognitive Level Weights. The cognitive level weights 
vector associated with the decision type is identified by a simple 


table look-up procedure. 


Select the Most Effective Instructional Method. A linear evaluation 
procedure is performed by a summation over the results of multipli- 
cation of elements of the cognitive levels weight vector by 
corresponding elements of each vector associated with different 
methods in the method/attribute relevancy matrix. This procedure 
provides an effective value assessment for each instructional 
method. The method with the highest effectiveness value will be 
selected as the most effective instructional method for training 

the decision task under consideration. 


Identify the Instructional Media Incompatible with the Selected Method. 
The vector associated with the selected instructional method in the 


method/media relevancy matrix is identified by a simple table 
look-up. Those instructional media corresponding to zero-value 
entries in the vector are considered as the ones incompatible with 
the selected method. 


Select the Most Effective Instructional Media. The degrees of 
importance of attributes, associated with each media, with respect 
to the training environment is defined in a weight vector. Then, a 
linear evaluation procedure, similar to the one used in item (Ff), 
is performed to select the most effective instructional media. A 
summation is performed on the results of multiplication of elements 
of the weight vector by corresponding elements of each vector 
associated with different media in the media/attribute matrix. The 
result of this summation will define the degree of effectiveness 

of the media if the media is compatible with the selected method. 
In the case that the media is not compatible the degree of effec- 
tiveness will be zero. The media with the highest degree of 
effectiveness will be selected as the most effective instructional 
method to be used with the selected instructional method in the 
specific instructional environment. 
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SECTION V 
CONCLUSIONS AND RECOMMENDATIONS 
GENERAL 


A system to generate instructional guidelines for a task-specific 
decision training program, in any training environment, was developed. The 
system selects the most effective instructional method and media and provides 
required instructional contents for any specific decision task, subject to 
training in a specific training environment. The project has been aimed 
not only to develop a systematic procedure for providing instructional 
guidelines for decision tasks encountered in LAMPS ASW operation, but also 
to design a task-independent methodology in order to make application of 
the system in other task domains subject to minimal modification effort. 

Such a methodology allows incorporation of general purpose decision training 
components into task specific decision training programs. This incorporation 
provides trainees with the basic skills required for making task-specific 
decisions within a new domain, in case of their transfer to other task domains. 
Furthermore, since the characteristics of decision tasks within any specific 
domain are subject to change with time, the methodology will allow development 
of more effective training programs even in the case where there is no 
possibility of transferring the trainees to other task domains. Finally, the 
methodology will provide for the carry-over of the decision making capability 
into incidental functions, which are usually left untrained. 


The instructional content was presented in two categories: task-specific 
information and basic instructional content. The task specific information 
is provided by decision trees resulting from a detailed analysis of the 
decision trees resulting from a detailed analysis of the decision tasks. The 
basic instructional content is generated through a classification of the 
decision tasks into one of the possible classes relevant to decision training. 
Instructional methods and media are selected by a quantified scheme which takes 
into account the characteristics of the training environment as well as the 
type of the decision tasks. 


The Instructional Guideline Generation System provides a systematic 
procedure for incorporating a great deal of data into the process of selecting 
different components of a decision training program. These data are of great 
importance to any instructional system development and, theoretically, should 
all be considered in the development of any program. The procedure of 
instructional guideline generation, developed in this project, provides 
systematic means for the management of data and its effective utilization in 
the process of instructional system development. However, a more effective 
utilization of the instructional guideline generation system requires a more 
detailed look into three problems: (1) elicitation of more accurate judgmental 
data, (2) a control structure to guide the use in executing different steps 
of the procedure, and (3) a computational aid for calculations required in 
the procedure. 


A computerized instructional guideline generation aid will provide a 


vehicle for analysis of all three problems. Such an aid and a plan for 
realization of it is addressed in the next section. 
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COMPUTERIZED INSTRUCTIONAL GUIDELINE GENERATION AID 


Effectiveness of the instructional guideline generation system can be 
highly enhanced by improvement of the system in three dimensions: 


a. elicitation of more accurate judgmental data 


b. generation of a control structure to guide the user in 
executing different steps of the system 


Cc. development of a computational aid for calculations required 
in the system. 


A computerized instructional guideline generation aid can result in substantial 
improvements in all three dimensions. 


The aiding system follows the design described in Chapter IV. It has 
the potential for working in two different modes: development mode and 
application mode. Development mode will provide access to judgmental data 
stored in the system. The accuracy of each element of the data will be 
assessed by an analysis of the guidelines generated for decision tasks used 
as test cases. The inaccurate data may be modified to adapt the system's 
output to desirable responses. Application mode represents a man/machine 
interactive session through which the information concerning the character- 
istics of the training environment, as well as the input decision task, is 
elicited from the user upon which the instruction guidelines for the decision 
task will be generated. During interactive sessions, the user will be 
guided by the sequence and content of a series of questions designed to 
elicit the required information. Furthermore, all calculations will be 
performed by the system without user's interaction. Such a system simplifies 
the process of instructional guideline generation into a series of judgmental 
evaluations of elementary attributes. While the process is highly simplified 
for the user, the system automatically incorporates all the information 
acquired from the user into the process which results in recommendations for 
the most effective instructional method and media as well as the required 
instructional contents. 


A prototype application mode session of the system is simulated in the 
following pages. The user responses have been underlined. 


Describe the characteristics of the instructional environment by 
assigning an important weight (between zero and one) to each of the following 
attributes. 


1. Learner paced control on the instructional media. 
AT(1) = .7 


2. Instructional media to be self instructional. 
AT(2) = .8 


3. Motion of the instructional media. 
AT(3) = .3 
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4. Fidelity of the instructional media in terms of situational details. 
AT(4) = .8 
5. Learner control over the instruction speed (motion). 
AT(5) = .6 
6. Flexibility of the media for instructional sequencing. 
AT(6) = .4 
7. Instruction readability. 
AT(7) = .3 
8. Context creation. 
AT(8) = .8 
9. Convenience for development of testing and evaluation. 
AT(9) = .6 
10. Low cost of instruction. 
AT(10) = .3 
11. Simplicity of use and maintenance. 
AT(11) = .9 
Here is the attribute list with the weights you assigned. 
1]. WEIGHT = 0.70: Learner paced control on the instructional media. 
2. WEIGHT = 0.80: Instructional media to be self instructional. 
3. WEIGHT = 0.30: Motion of the instructional media. 
4, WEIGHT = 0.80: Fidelity of the instructional media in terms of situation. 
5. WEIGHT = 0.60: Learner control over the instruction speed (motion). 
6. WEIGHT = 0.40: Flexibility of the media for instructional sequencing. 
7. WEIGHT = 0.30: Instruction readability. 
8. WEIGHT = 0.80: Context creation. 
9. WEIGHT = 0.60: Convenience for development of testing and evaluation. 
10. WEIGHT = 0.30: Low cost of instruction. 
11. WEIGHT = 0.90: Simplicity of use and maintenance. 
Options are Type 
1. Weights are satisfactory. q 
2. Change weight of an attribute. Cc 
3. Display complete list. d 
OPTION = c 
10. WEIGHT = 0.30: Low cost of instruction. 
OPTION = g 
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Here is the attribute list with the weights you assigned 


1. WEIGHT = 0.70: Learner paced control on the instructional media. 

2. WEIGHT = 0.80: Instructional media to be self instructional. 

3. WEIGHT = 0.30: Motion of the instructional media. 

4. WEIGHT = 0.80: Fidelity of the instructional media in terms of situation. 
5. WEIGHT = 0.60: Learner control over the instruction speed (motion). 
6. WEIGHT = 0.40: Flexibility of the media for instructional sequencing. 
7. WEIGHT = 0.30: Instruction readability. 

8. WEIGHT = 0.80: Context creation. 

9, WEIGHT = 0.60: Convenience for development of testing and evaluation. 
10. WEIGHT = 0.50: Low cost of instruction. 

11. WEIGHT = 0.90: Simplicity of use and maintenance. 


Throughout this consultation session the above importance weights will be 
considered to represent the elements of the instructional environment. To 
present a new environment cancel the current session and start a new one. 
Press RETURN when ready to continue. 

Introduce the new instructional task by its instructional objective. 


Enter description: press the ESC key then RETURN. 


Classify the Contact as U.S. Ship, Enemy Ship, or Natural Object. 


Do you want to change or correct the description? Type yes/no. No 


What is the time pressure on course representation? 
Specify it by a time pressure factor between zero and one. 


Factor = .3 


Does the objective of the task include selection of an alternative from a 
set of alternatives? Yes 


Is the set of alternatives given? Yes 


Is there a completely specified procedure for the alternative selection? 
No 


Once the alternatives are given, is there any critical time pressure on the 
operator for fast alternative selection? Yes 


Which of the following decision rules is the most suitable one for the task? 


Subjective Expected Utility (SEU) 
Mini-max loss (MMX) 

Mini-min loss (MMN) 

Mini-max regret (MMR) 

Lexicograph (LEX) 

Hurwicz (HUR) 

Satisficing (SAT) 


NOP WNh— 


Enter a number between 1 and 7? 2 
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INSTRUCTIONAL GUIDELINES 


Instructional objective: classify the contact as U.S. Ship, Enemy Ship, or 
Natural Object. 


The task to be taught is a decision task. 

Decision Type: Type 2 

Decision Class: 2-PMMX 

Instructional Content Must Include the Following Items: 


Collecting subjective utilities under time pressures. Translating outcomes 
into utilities. Aggregating utilities. Relation of loss and utility 
concept of minimizing maximum loss. Effect of time pressure on quality. 
Concept of payoff tables. 


The Best Three Suggested Instructional Methods are: 


1. Discussion-Individual Tutorial for which the best three 
instructional media are: 


1. Programmed Text 
2. CAI 
3. Lecture (oral presentation/chalkboard) 


2. Individual Investigation (Research) for which the best three 
instructional media are: 


1. Natural Objects 
2. Textbook 
3. Workbook, problem sets 


3. Projects - Individual for which the best three instructional media 
are: 


1. Natural objects 
2. Textbook 
3. Workbook, problem sets 


Press RETURN when ready to continue. 
EVALUATION PLAN 


Two general evaluation areas should be reviewed in assessing the impact 
of the decision training guidelines described earlier in this report. They 
are: (1) evaluation of the impact on training system design, development 
and implementation, and (2) evaluation of the impact on trainees. It should 
be noted that as one proceeds from the initial instructional design stages 
to ultimate field performance more and more intervening variables enter into 
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‘the picture. The topics identified below are presented with full recognition 
that the decision training guidelines are only one of several important 
variables which effect the overall instructional systems process. 


a. 


Design. The design stage refers to the initial phase of 
Instructional system development in which decision tasks 

are identified from among the full set of tasks which 
constitute LAMPS operator functions. At this stage a general 
approach to subject matter and instruction is defined. 
Relevant evaluation questions at this stage involve the 
areas of accuracy, complexity, time/effort, and expertise. 


Does the classification scheme accurately identify all 
tasks which are in fact decision tasks? What modifica- 
tions are required to improve the accuracy of classification? 


Can the guidelines be applied easily, rapidly? 


What level of expertise in behavior decision theory is 
required to apply the guidelines. 


What level of expertise in instructional methods/media 
is required to apply the guidelines. 


Is the information provided to the instructional designed 
sufficient to develop a general approach to instruction 
which integrates all the decision tasks to be taught? 


Development. Once decision tasks have been identified and 


characterized in terms of relevant decision theoretic and 
instructional guidelines, instructional materials are 
developed and tested. Questions of interest in the 
development stage involve the completeness and robustness 
of the guidance information provided. 


To what extent do the supporting conceptual materials 
meet the developer's needs for subject matter guidance? 
Conversely, how much time/effort is required to supplement 
content guidelines? 


To what extent do the guidelines provide direction to the 
developer in selection of an instructional strategy? 


What are the effects of environmental factors such as 
instructional setting and trainee groups on the utility of 
the content/instructional guidelines? How robust are the 
guidelines in the face of variations in trainee groups, 
training settings? 


How much assistance do the guidelines provide when making 


revisions to draft instructional materials subsequent to 
testing? 
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Implementation. This phase of instructional systems development 
includes the installation and management of the instructional 
system. It will be of interest at this stage to determine what 
problems exist in the areas of logistics and support, acceptance, 
instructor skills, management, and integration with related (non- 
decision) LAMPS training. 


What problems of acceptance of the instructional system 
occur for students, instructors? 


Are special skills in instructional technique or expertise 
in subject matter required of instructors? 


Does management of instruction pose special problems 
of data handling or analysis? 


Is decision training complementary to and supportive 
of the content of related (non-decision) LAMPS training? 


Training Efficacy -- Didactic Instruction. At this stage of the 
instructional systems development process, full scale classroom 
instruction occurs. Major evaluation questions at this point 
deal with the level of attainment of training objectives by 
trainees, and the associated costs and time required for 
attainment of objectives. 


Do all trainees achieve minimally acceptable levels of 
knowledge and skill as defined in course objectives? 

What areas of weakness, strength can be identified for 
individual students? For the trainee group as a whole? 


What costs and how much time are associated with attainment 
of individual objectives? With completion of the course as 
a whole? 


Training Efficacy -- Training Missions. In this stage, the 
LAMPS student pilot is exposed via training missions to a 
variety of in-flight situations some of which may test his 
decision skills as well as provide an opportunity for direct 
decision task instruction by a LAMPS instructor. The evaluation 
topics identified in the previous section apply here to any 
in-flight decision task instruction which may occur. In 
addition, the amount of transfer from the didactic situation 

is of interest to determine. 


What benefits (detriments) to decision task performance 
during in-flight training can be identified (for students 
who have had didactic decision training)? 


Has in-flight training been enhanced in terms of time/costs 
required as a result of transfer from the didactic situation? 
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(f) Post-Training Performance. Satisfactory on-the-job performance, 
Subsequent to instruction, represents. the end-point of the 
instructional system. Questions of interest here relate to the 
impact of the instructional system on long term performance of 
LAMPS pilots. 


What evidence exists that decision training has improved 
mission effectiveness or safety? 


Can general improvements (deficiencies) in decision 
performance as a result of decision training be demonstrated 
or are the benefits (detriments) restricted to specific 
tasks/situations for which training was provided? 


What is the retention level of decision skills three months 
after training? After one year? After two years? 


What major content deficiencies in pilot decision skills 
remain? What modifications to the content coverage of 
formal decision training seems necessary? 


Acquisition of data to answer all of the evaluation questions identified 
above will involve a series of evaluation efforts, some formal and some 
informal. Table 21 summarizes the principal evaluation questions to be 
answered by stage together with an identification of sources/methods for 
acquiring necessary data. Much of the evaluation information identified will 
be of use in revision, either of the decision training guidelines or of the 
decision training itself. That is, formative use of the evaluation data is 
more likely to occur and will probably be of more benefit than summative 
evaluation, the overall assessment of the value of the guidelines and/or 
training. Summative evaluation will be difficult to carry out for a number 
of reasons, two of which are particularly important. First, training 
settings typically do not lend themselves to a true experimental or even a 
quasi-experimental analysis of instructional impact. Control of internal 
and external threats to validity (e.g., mortality, selection, history, 
maturation, testing, reactive effects, testing-training interaction, etc.) 
require experimental designs which usually depend on more freedom in selection 
of trainees and treatment conditions that the operational training setting will 
permit. Second, the time, costs, preserverance and cooperation required to 
obtain representative and comprehensive post-instructional performance data 
are great and, as a result, less useful proxy measures such as performance 
at time of exit/graduation from training are typically accepted as substitutes. 


In spite of these considerations, Table 21 presents an overview of 
an idealized approach to evaluation in order to document the full scope of 
evaluation analysis which could be brought to bear. The actual evaluation 
effort that is mounted should represent a compromise between the value of 
the information sought for each of the six general areas identified and the 
cost and difficulty of obtaining it. 
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TABLE 21. OVERVIEW APPROACH FOR EVALUATING 
DECISION TRAINING GUIDELINES 


STAGE EVALUATION TOPIC SOURCE OF DATA/METHOD COMMENT/LIMITATIONS 
Design Accuracy of Comparison to expert Inter-judge reliability 
classification judgment should be demonstrated 
Ease of Time, costs expended 
guidelines by designer 
Expertise required Identification of Dependent on skill 
to use guidelines decision tasks or of course designer 
instructional 


methods which pose 
problems for 
designer 


Generality of Opinion of designer, Subjective 
approach afforded developer 
by guidelines 


Develop- Adequacy of Identification of Dependent on skill 
ment guidelines areas in which of developer 

additional subject 

matter or instruc- 

tional information 

is requires; include 

measures of time spent 

supplementing guide- 

lines by developer 


Robustness in Similar to above Subjective 
face of varying 

training settings, 

trainee groups 


Assistance in Opinion of developer Subjective 
making revisions 
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STAGE 


Imple- 
menta- 
tion 


Didactic 
Training 


Training 
Missions 
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TABLE 21. OVERVIEW APPROACH FOR EVALUATING 


DECISION TRAINING GUIDELINES (CONTINUED) 


EVALUATION TOPIC 


Coordination with 
related training 


Acceptance 


Instructor skills 
required 


Management of 
instructor 


Integration with 
related training 


Students achieve- 
ment levels 


Costs, time 


Transfer from 
didactic train- 
ing 


SOURCE OF DATA/METHOD 


Identification of 
unusual. logistical 
problems or require- 
ments 


Opinions of students, 
instructors 


Opinions of 


instructors 


Special data handling/ 
analysis requirements 


Opinions of students, 
instructors 


Individual and group 
achievement of ob- 
jectives (by objective 
and for course as a 
whole) 


Same as above 


Performance of 
students on decision 
tasks encountered 
in-flight 
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COMMENT/LIMITATIONS 


Dependent on training 
site chosen 


Subjective 


Dependent on skills 
of instructors chosen; 
subjective 


Subjective 


Requires well defined, 
comprehensive set of 
objectives; existence 
of an alternative 
treatment group com- 
parison is unlikely; 
true-experimenta 1 
design may be hard 

to implement in op- 
erational setting 


Same as above; should 
include presentation 
costs as well as 
trainee costs 


See comments under 
"DIDACTIC TRAINING" 


STAGE 


Post- 
Training 
Perfor- 
mance 
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TABLE 21. 


OVERVIEW APPROACH FOR EVALUATING 


DECISION TRAINING GUIDELINES (CONTINUED) 


EVALUATION TOPIC 


Reduced costs, 
time of in-flight 
training 


Improved mission 
effectiveness 


Generality of 
effects 


Retention of 
decision skills 


Course areas 
requiring 


SOURCE OF DATA/METHOD 


Number of missions 
missions required to 
achieve proficiency 


Critical incidents, 
pilot opinion, 
mission logs 


Pilot opinion 


Test trainees 
periodically 
after graduation 


Critical incidents, 
pilot opinion, mission 
logs, periodic tests 
of graduates 
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COMMENT/LIMITATIONS 


See comments under 
"DIDACTIC TRAINING" 


Difficulty of 
collecting operational 
data and subjective 
limitations 


See above 


Follow-up difficult 
to implement 


Combination of infor- 
mation from three 
categories above 
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DECISION AIDS 


Analysis of the possible roles of a human operator as a decision maker 
identified two different roles: an operator as a decision maker and an 
operator as a decision executor. Although these two roles may be played by 
the same person, it is also possible that different individuals be responsible 
for each of the two roles. In this case, an individual makes the decision by 
providing the decision solutions which identifies the best couse of action to 
be taken under each possible environmental condition. The decision solution 
is then communicated to the second person who acts as a decision executor. 

At any decision time, the decision executor detects the current environmental 
condition and implements the decision solution corresponding to the detected 
condition. The communication media by which the information concerning 
decision solution is transmitted from the decision maker to the decision 
executor constitute the decision aid (Figure 21). 


Analysis of Pilot/ATO's and Sensor Operator's jobs suggests the need 
for decision aids in critical decision situations. In many critical decision 
tasks there is not enough time for a reasonable decision making; furthermore, 
the knowledge of the criticality of the situation might further complicate 
any on the job decision making. Meanwhile, the operator can not afford any 
non-optimal decisions. A decision aid, in such a situation, will provide the 
operator with optimal solutions which have been provided in advance and under 
no time pressure. 


In this context the term decision aid has a very broad meaning. Decision 
aids can range from pre-programmed "look-up" matrices and tables to computer 
driven displays. Some examples of decision aids are as follows: (1) Non-computer 
oriented. Tables and matrices, blank standardized forms to be filled out 
that structure the decision and its data, grease pencils and plastic status 
boards, warning lights in the cockpit, and the like, (2) Computer oriented. 
Visual displays and computer graphics, automatic warnings, CRT displays, 
adaptive decision aiding (e.g., ADDAM), and computerized sequence of interaction 
decision making, information entry, aggregation and display, and so on. Each 
decision aid can itself be characterized in terms of a number of attributes 
such as 


Maintainability 
Cost 
Completeness of aiding (disaggregation) 
Turn-around time 
Amount of manual input required 
Complexity of the information display 
For example, a simple cockpit warning light provides very rapid and very 


aggregated decision aiding. Either the light is on or it is off. On the 
other hand, an interactive, computer based system that requires input from 
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POSSIBLE 
ENVIRONMENTAL 
CONDITIONS 
DECISION DECISION 7 | 
See _____g «DECISION 
MAKING - MAKER SOLUTIONS 


TASK 


DECISION 


EXECUTOR SOLUTION 


DECISION 
, ——— IMPLEMENTATION 


SOLUTIONS 


CURRENT 
ENVIRONMENTAL 
CONDITION 


Figure 21. The Role Of Decision Aids In 
A Decision Making Process 
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a decision executor takes time, but it is more complete in its display 

(and also more complex than a single light). Prepackaged check lists are 
mid-way in terms of turn around time, complexity and input information 
required. In summary, each decision task can be supported with a decision aid 


and each decision aid can be developed with the desired aid characteristics 
in mind. 
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APPENDIX A 
ANALYSIS OF OBJECTIVE HIERARCHIES FOR PILOT/ATO 


Struc- Decision 
ture Page Task No. Type Keyword Task Area 
2 1.1.7 Type 2 Classify, specify Contact Classification 
[ the degree 
+ 47 1.1.7.3 Type 2 Determine Whether 
3. 1.1.1.4 Type 2, Determine What, Equipment to Carry 
Type 2 Determine What 
3? VF ekks5 Type 1 State Flying Methods 
ie 3. 1.1.1.6 Type 2 Determine whether Mission safety 
3. 1.1.17.6.1 Type 1 State all 
Ee 3. 1.1.1.6 Type 2 Determine whether Mission safety 
3. 1.1.1.6.4 Type 2 Determine whether 
3. 1.1.1.7.1 Type 1 State Crew Briefing 
14 «1.1.4.3.1 Type 3 Recognize Malfunction Recognition 
a 16 «1.1.4.3.1.1 Type 3 Recognize : 
16 «1.1.4.3.1.1.1 - Type 2 Indicate which 
16 «1.17.4.3.1.17.7.1 Type 1 Describe the 
components 
> V7 1.1.4.3.1.1.1.1.2 Type 2 State the likely 
effect 
iz 14 -1.1.4.3.2 Type 3 Recognize Malfunction Recognition 
> 18 1.1.4.3.2.1.1 Type 2 Indicate which 
“14 1.1.4.3.3 Type 3 Recognize Malfunction Recognition 
[ 19 «-1.1.4.3.3.1.1 Type 2 Indicate which 
[ 14 =1.1.4.3.4 Type 3 Recognize Malfunction Recognition 
20 =«1.1.4.3.4.1.1 Type 2 Indicate which 
C 14 1.1.4.3.5 Type 3 Recognize Malfunction Recognition 
2] 1.1.4.3.5.1.1 Type 2 Indicate which 
r 14 -1.1.4.3.6 Type 3 Recognize Malfunction Recognition 
> 22 1.1.4.3.6.1.1 Type 2 Indicate Which 
fe 14 -1.1.4.3.7 Type 3. Recognize Malfunction Recognition 
23. «1.1.4.3.7.1.1 Type 2 Indicate which 
is 14 = «*1.1.4.3.8 Type 3 Recognize Malfunction Recognition 
24 -1.1.4.3.8.1.1 Type 2 Indicate which 
14 «21.1.4.3.9 Type 3. Recognize Malfunction Recognition 
. is 25. -1.1.4.3.9.1.1 Type 2 Indicate which 
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Struc- Decision 
ture Page Task No. Type Keyword Task Area 
e 14 =1.1.4.3.10 EType 2 State, most likely Type of Fire 
27. 1.1.4.3.16.1 Type 1 State 
ie 14 -1.1.4.3.10 EType 2 State, most likely Type of Fire 
27): 1.1.4.3.10.3 Type 1 State’ 
14. 1.1.4.3.10 ZType 2 State most likely Type of Fire 
27) -1.1.4.3.10.5 EType 2 State the most likely 
28 s«1.1.4.3.10.5.2 EType 2 State the likely 
effect 
29. «1.1.4.3.11.1.2 Type 2, Determine the Mission Urgency 
Type 2 urgency, if 
29. «1.1.4.3.11.3 Type 3 Recognize Electric Failure 
Recognition 
31. -1.1.4.3.12.1.1 Type 2 Indicate which Malfunction Recognition 
32. -1.1.4.3.12.3.2 rType 2 State likely effect Fuel System Failure 
33. «1.1.4.3.13.1.1 Type 2. Indicate which Tire Blow-out 
34 1.1.4.3.14.1.1 Type 2 Indicate which Tire Blow-out 
35. -1.1.4.3.15.1.1 Type 2 Indicate which Loss of Tail Rotor 
Control 
~ 38 1.1.6.1 Type 3. Choose an appro- ASW Mission Tactic 
| priate 
L 38 1.1.6.1.1 Type 2 Determine the likely 
| identity 
ib 39)s«d.1.6.1.1.5 Type 2 Determine the likely 
identity 
ww 38° eG al Type 3. Choose an appro- ASW Mission Tactic 
priate 
L. -38° «1.1.6.1.2 Type 1 Determine possible 
| behavior 
- 40 1.1.6.1.2.1 Type 3 Determine the most 
likely 
42 1.1.6.1.2.2.3.1.2.3.4 Type 2 Classify Sound/Velocity Profile 
Interpretation 
- 38 1.1.6.1 Type 3 Choose an appro- ASW Mission Tactic 
| oriate 
[* 38 —«1.1.6.1.3 Type 1 State 
45 1.1.6.1.3.3 Type 2 Determine the pro- 
portion 
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Struc- 


ture Page 


yt 1 


38 


51 


53 
53 


55 
56 
57 


55 
56 
58 


55 
56 
59 


55 
56 
60 


. 


Task No. 
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Decision 


Type 
Type 


Type 
Type 


Type 


Type 
Type 


Type 
Type 


Type 
Type 
Type 
Type 


Type 
Type 


Type 


Type 
Type 


Type 
Type 


IType 2 


Type 
Type 


tType 2 


Type 
Type 


LType 2 


Type 
Type 


IType 2 


3 


] 
2 
3 


Www rw no— 


2 


3 
3 


3 
3 


3 
3 


3 
3 


Keyword 


Choose an appro- 
priate 

State 

Determine the depth 


Choose an appro- 
priate 

State 

Determine if 


Revise plan 
Determine whether 


Revise plan 
Select an appro- 
priate 


Determine if 


Prescribe an 
appropriate 


Determine which 


Determine an 
appropriate 


Select an appro- 
priate 


State the proper 
State the proper 


Identify 
Recognize 
Indicate which 
accompany 


Identify 
Recognize 
Indicate which 


Identify 
Recognize 
Indicate which 


Identify 


Recognize 
Indicate which 
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Task Area 


ASE Mission Tactic 


ASW Mission Tactic 


Tactic Revision 


Tactic Revision 


Pattern Expansion 


Pattern Expansion 


Sonobuoys Monitoring 


Pattern Expansion 


MAD Tracking Pattern 


Refueling 
Refueling 


Engine Malfunction 


Engine Malfunction 


Engine Malfunction 


Engine Malfunction 


Struc- 
ture Page 


b 


Task No. 


. 


. 


NAVTRAEQUIPCEN 77-C-0005-1 


Decision 
Type 
Type 3 
Type 3 
LType 2 
Type 3 
Type 3 
rType 2 
Type 3 
Type 3 
ZType 2 


EType 2 


Type 3 


Type 3 
IType 2 


Type 3 


Type 3 
Type 2 


Type 3 


Type 3 
EType 2 


rType 2 


Type 3 


Type 3 
LType 2 


Type 3 


Type 3 
rType 2 


Type 3 


Type 3 
Type 2 
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Keyword 
Identify 
Recognize 
Indicate which 
Identify 
Recognize 
Indicate which 
Identify 
Recognize 
Indicate which 


State the likely 
effects 


Identify 


Recognize 
Indicate which 


Identify 


Recognize 
Indicate which 


Identify 


Recognize 
Indicate which 


State the likely 
effect 


Identify 


Recognize 
Indicate which 


Identify 


Recognize 
Indicate which 


Identify 


Recognize 
Indicate which 


Task Area 


Engine Malfunction 


Engine Malfunction 


Engine Malfunction 


Engine Malfunction 


Drive System 
Malfunction 


Drive System 
Malfunction 


Drive System 
Malfunction 


Transmission 
Malfunction 


Electrical System 
Malfunction 


Electrical System 
Malfunction 


Electrical System 
Malfunction 


Struc- 
ture 


mn orm oe es 2 | 


~] 


¥ 


Page 
55 


70 
74 


55 


75 
76 


55 


75 
77 


55 


75 
78 


79 


80 


Task No. 
.10.1.3 
.10.1.3.4 
-10.1.3.4.1.1 
.1.10.1.4 
~1.10.1.4.1 
.10.1.4.1.1.1 
-1.10.1.4 
»1.10.1.4.2 
.10.1.4.2.1.1 
.10.1.4 
-1.70.1.4.3 
.10.1.4.3.1.1 
.10.1.4.4.2 
-10.1.4.5.2 
-10.1.5 
.10.1.5.1 
~10.1.5.1.17.1 
-10.1.5 
.10.1.5.2 
~10.1.5.2.1.1 
.10.1.5.3.2 
-10.1.5.4.2 
-10.1.5.5.2 
.10.1.6 
-10.1.6.1 
-10.1.6.1.1.1 
-10.1.6.2.2 
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Decision 
Type 


Type 3 


Type 3 
rType 2 


Type 3 


Type 3 
rType 2. 


Type 3 


Type 3 
rType 2 


Type 3 


Type 3 
rType 2 


rType 2 


rType 2 


Type 3 
Type 3 
xrType 2 
Type 3 
Type 3 
rType 2 


rType 2 


rType 2 


rlype 2 


Type 3 


Type 3 
LType 2 


rType 2 


Keyword 
Identify 


Recognize 
Indicate which 


Identify 


Recognize 
Indicate which 


Identify 


Recognize 
Indicate which 


Identify 


Recognize 
Indicate which 


State the likely 
effects 


State the likely 
effect 


Identify 
Recognize 
Indicate which 


Identify 
Recognize 
Indicate which 


State the likely 
effect 


State the likely 
effect 


State the likely 
effect 


Identify 


Recognize 
Indicate which 


State the likely 
effect 
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Task Area 


Electrical System 
Malfunction 


Fuel System 
Malfunction 


Fuel System 
Malfunction 


Fuel System 
Malfunction 


Engine 071 System 
Malfunction 


Transmission Oil System 
Malfunction 


Oil System Malfunction 


O11 System Malfunction 


Main Rotor Malfunction 
Brake System 
Malfunction 


Blade Track System 
Malfunction 


Flight Controls 
Malfunction 


Hydraulic Power Supply 
Malfunction 


Struc- 
ture Page 


55 
90 
92 
93 


95 
97 
98 
99 
100 
101 
102 
104 
106 
107 


107 
107 


107 
107 
108 
109 
110 
111 
1 
112 
112 


Task No. 


etal; 
hy 10, 


. 


] 


2. 


rs Pera ee 
ape 
1. 
eaves 


. . ° 
> > Ww 


Decision 
Type 


Type 3 
Type 2 
Type 2 
Type 2 


Type 2 
Type 2 
Type 2 
Type 2 
Type 2 
Type 2 
Type 2 
Type 2 
Type 2 
Type 2 


Type 2 
Type 3 


Type 3 
Type 1 
Type 2 
Type 2 
Type 2 
Type 2 
Type 2 
Type 2 
rType 2 
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Keyword 
Identify 
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Identify those 


Identify whether 


Determine 
Determine 
Determine 
Determine 
Determine 
Determine 
Determine 
Determine 
Determine 
Determine 


Determine 
State the 


Determine 
State the 
native 

Determine 


Determine 


Determine i 


Determine 


Determine i 


Determine 


State the 


Determine the type 
(known possibilities) 


whether 
it 
if 
if 
if 
if 
whether 
if 


whether 


the min... 


whether 
criteria 


which 
alter- 


if 


whether 


cues 


Task Area 
Hydraulic System 
Malfunction 
Landing Methods 


Engine Failure 


Landing Feasibility 
Land or Abort 

Land or Abort 

Land or Abort 

Land or Abort 

Land or Abort 

Land or Abort 
Engine Failure 
Landing Feasibility 
Flying Method 

Land or Wave Off 


Landing Method 


Landing Safety 
Landing Safety 
Landing Safety 
Flying Method 
Landing Safety 
Land or Abort 


Loss of Tail Control 


Struc- 
ture Page 


Task No. 


114 «1.7.12.1.3. 


115 -1.1.12.1.3. 


116) «-1.1.12.1.3 
116 -1.1.12.1.3 


116 «1.1.12.1.3 
116 -1.1.12.1.3 
116—=«1.1.12.1.3 


118 «-1.1.13.2 
118) -1.1.13.3 
118 =«1.1.13.4 
120. «1.2.6.3 
120. «1.2.6.4 


[ 120 «1.2.6.5 
120 «-1.2.6.5.1 


r 120 1.2.6.5 
120. 1.2.6.5.2 


120 «(1.2.6.6 

120 «1.2.6.7 

123. :1.3.5.4.2 
123° -1.3.5.4.3 
123. «1.3,.5.4.4 
128 «1.4.6.1.1. 
128 «1.4.6.1.1. 


4.1.2 
4.2.1 


4.3.1 
4.4.1 
24.5.1 


2 
6.2 
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Decision 

Type. Keyword 
Type 2. Determine if 
Type 2 Determine if 
Type 3. Identify 
Type 3. Recognize 
Type 3. Recognize 
Type 3. Recognize 
Type 3. Identify 
rType 3 State 

rType 3. State 

rType 3 State 

Type 2. Determine 
Type 3. Determine the 

appropriate 
Type 3, Determine, 
Type 3. determine 
Type 3, Determine, 
Type 3, determine, 
Type 3. determine 
Type 3, Determine, 
Type 3. determine 
Type 3, Determine, 
Type 3. determine 
rIType 3 State the appro- 
priate ref's 

Type 3. State 

Type 3. Classify 
Type 3. Classify 
Type 3. Classify 
Type 3. Determine which 
Type 1 State 
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Task Area 
Postflight Inspection 
Postflight Inspection 
Tail Wheel Lock Failure 


Total Electrical 
Failure 


Droop Stop Faiiure 
Internal Engine Fire 
External Engine Fire 
Intelligence Gathering 
Tactics 

Crew Briefing 

Tactics 


Tactics 


Tactics 


Tactics 


Tactics 


Tactics 
Tactics 
Tactics 
Tactics 


Search and Rescue 
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Struc- Decision 
ture Page Task No. Type Keywords Task Area 

128 «1.4.6.1.1.3 _Type 2 Determine the best Search and Rescue 

128 «=421.4.6.1.1.4 Type 3 State Search and Rescue 

129. «+1.4.6.1.3.1.1 Type 3 Identify Search and Rescue 

129 +1.4.6.1.3.3.1.1 Type 3 State the cues Search and Rescue 

[ 11: 146202 Type 3. Determine which Search and Rescue 

131 «1.4.6.2.1.2.1 Type 1 State the major 

132 «1.4.6.2.2.2 Type 3. Determine which Flying Method 

132) «1.4.6.2.2.3 Type 1 State the major Flying Method 

133) -1.4.7.1.2.2 Type 2 Determine Landing Feasibility 

134) «-1.4.7.2.3.1 Type 2 Determine Search and Rescue 

135 -1.4.7.3.1.1.1.2 Type 3. Locate Search and Rescue 

136. Ve4e7oele Type 3. Identify Doppler Failure 

136 «1.4.7.5.2.1 Type 3. Identify Radar Altimeter Failure 

139 1.5.5.1.6 Type 2 State all appro- External Cargo 
priate 

140 =1.5.5.3.3 Type 2 State all appro- External Cargo 
priate 

142 =+1.6.1.1 Type 3. Determine Internal Cargo 

142 «1.6.1.2 Type 3. State Internal Cargo 

145 «1.7.5.5 Type 3. Determine Gunfire Spotting 

150 =1.8.2.2.4 rType 2 State likely Lighting System 
effects Failure 

152. 1.8.2.3.1.2 LType 2 State likely Flight Instrument 
effect System Failure 

153) -1.8.2.3.2.2 rType 2 State likely AN/AYK-2 System 
effect Failure 


128 


Struc- 

ture Page Task 
155. | 7852.3; 
159 «1.8.2.3. 
160 1.8.2.3. 
163) «1.8.2.4. 
164 1.8.2.4. 
165 1.8.2.4. 
166 «1.8.2.4. 
167. 1.8.2.4. 
168 1.8.2.4. 
171 (1.8.2.5. 
174 (1.8.2.5. 
lee’, <WeB.Ge3- 


NAVTRAEQUIPCEN 77-C-0005-1 


Decision 
No. Type 
4.2 ErType 2 
8.2 rType 2 
9.2 EType 2 
1.4 rType 2 
2.2 rType 2 
3.2 ZrType 2 
6.2 Type 2 
7.2 ZType 2 
8.2 ZType 2 
2.2 zrType 2 
5.2 ZType 2 
2 ZType 2 


Keywords 


State likely 
effects 


State likely 
effects 


State likely 
effects 


State likely 
effects 


State likely 
effects 


State likely 


effects 


State likely 


effects 


State likely 
effects 


State likely 
effects 


State likely 
effects 


State likely 
effects 


State likely 
effects 
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Task Area 


Standby Compass 
Failure 


AN/APN-182 System 
Failure 


AN/ASN-50 System | 
Failure 


Sonobuoy Processing 
System Failure 


Smoke Maker Dispenser 
System Failure 


Torpedo Resease 
System Failure 


Sonobuoy Launcher 
System Failure 


MAD System Failure 
ALR-54 System Failure 
AN/ARC-159 System 


Failure 


Juliet 28 System 
Failure 


Anti-Ice System 
Failure 


Struc- 
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locke) foemen) loom) w Nw OW Ow 


0D 


fo) ao looken) an 


NMP 


Nh MP 


Task No. 
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PO PM DO PO 


PO PO 


RO Po 


RO PO 
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APPENDIX B 
ANALYSIS OF OBJECTIVE HIERARCHY FOR SENSOR OPERATOR 


Decision 


Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 
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Keyword 


Determine 
Determine 


Determine 
Determine 


Determine 
Determine 


Deternine 
Determine 


Determine 
Determine 


Determine 
Determine 


Determine 
Determine 


Determine 
Determine 


Determine 


Determine 
Determine 


Determine 
Determine 


Determine 
Determine 


Determine 
Determine 


Determine 
Determine 


Determine 
Determine 


Determine 
Determine 


Identify 


if 
if 


if 
if 


if 
if 


if 
if 


if 
if 


if 
if 


if 
if 


if 
if 


f 


— 


if 
if 
if 
if 
if 
if 
if 
if 
if 
if 
if 
if 
if 
if 


Task Area 


Daily Inspection 


Daily Inspection 


Daily 


Inspection 


Daily Inspection 


Daily 


Daily 


Daily 


Inspection 


Inspection 


Inspection 


Daily Inspection 


Fuel Acceptability 


Turnaround 


Turnaround 


Turnaround 


Turnaround 


Turnaround 


Turnaround 


Turnaround 


Inspection 


Inspection 


Inspection 


Inspection 


Inspection 


Inspection 


Inspection 


Standard Safety 


Precaution 
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6 
13 
4] 


42 
13 
4] 
42 
13 
4] 
42 
13 
4) 
42 
13 
4] 
42 
13 
4] 


42 


‘42 
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DOM Pho PM 
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NN 
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fon) 
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NM 
— 
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Fae ee | 


Type 
Type 
Type 


Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 
Type 


Type 


13] 


Decision 
Type 


2 
2 
2 
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Keyword 


Determine if 
Classify 


Perform all classi- 
fication functions 
to identify 
Classify 


Classify 

Perform all classi- 
fication functions 
to identify 
Identify 


Classify 

Perform all classi- 
fication functions 
to identify 
Identify 


Classify 

Perform all classi- 
fication functions 
to identify 
Identify 


Classify 

Perform all classi- 
fication functions 
to identify 
Identify 


Classify 

Perform all classi- 
fication functions 
to identify 
Determine as to 


. Determine if 


Classify 

Perform all classi- 
fication functions 
to identify 
Identify 


Task Area 


Turnaround Inspection 


Tactics, Target 
Classification 


Target Classification 


Target Classification 


Target Classification 


Target Classification 


Target Classification 


Target Classification 


Struc- 


ture Page 


as 


ol 


y 


mn 


NOP 


. 


Do PM 


poe ae) 


Task No. 


62.1.7 
ra ee 


nmr 


ine) 
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DO Po 
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Decision 


Type 


Type 
Type 


Type 
Type 
Type 
Type 
Type 
Type 
Type 


Type 
Type 


Type 
Type 


Type 
Type 


Type 


“Type 3 


Type 


Type 
Type 


xType 3 


Type 
Type 
Type 
Type 


aType 3 


Type 
Type 


3 
2 


NO Po 


2 
3 


3 


2 


3 


4 


3 
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Determine 1 


Keyword 
Classify 
Perform all classi- 
fication functions 
to identify 
Report probability 
Determine the Prob. 
Classify 
Perform all classi- 
fication functions 
to identify 
Classify 
Classify 


Determine if 


Determine i 


Determine 7 
Determine i 


Determine if 
Determine 


Determine if 
Determine 


Determine 


Determine 
State if 


Identify each 
Deternine 
Identify 
Determine 
Identify 
Determine 

State the effects 


Determine 


Task Area 


Target Classification 


Target Classification 


Flight Gear Inspection 


Flight Gear Inspection 
Flight Gear Inspection 
Mission Safety 
Mission Safety 


Interior Inspection 


SAR Inspection 


Prestart Inspection 
Prestart Inspection 
Radar Operation 

Duration Estimation 
Elevation & sistnuction 
Magnetic Bearing 
MAD Operation 


RO-32 Recorder 
Malfunction 


Struc- 
ture Page 


33 


33 
34 
34 
34 
36 
36 
36 
37 
39 
46 


47 * 
47 
47 
47 


48 
55 


55 
55 


55 


Task No. 
2.1.6.2.5 


Decision 
Type 


Type 3 


Type 
Type 
Type 
Type 


Oo WO Ww Ww W 


Type 
Type 
Type 
Type 
Type 
Type 3 
Type 3, 
Type 3 
Type 3, 
Type 3 
Type 2 
Type 2 


rType 2 
rType 2 


ZType 2 
LType 2 


rType 2 
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Keyword 


Determine 


Identify 

State 

Identify 
Identify 

State 

State 

State 

State 

State the best 
Identify 


Determine, select 


Determine, select 


Determine 


Determine 


Determine 


List all info. 
pertaining . 


List all info. 
pertaining 


List all info. 
pertaining 


List all info. 
pertaining 


Task Area 


ASQ-81 System 
Malfunction 


Tactics, Initial Contact 
ASQ-81 System Operation 
ASQ-81 System Operation 
ASQ-81 System Operation 
LN-66 Radar Operation 
LN-66 Radar Operation 
LN-66 Radar Operation 
ASA-26 System Operation 
Visual Search 


Tactics, Tracking with 
MAD 


Tactics, Master Sono- 
buoy Selection 


Tactics, Slave Sonobuoy 
Selection 


Percentage of 
Attenuation 


Percentage of 
Attenuation 


Range of Target 
SAR Mission 


SAR Mission 
SAR Mission 


SAR Mission 


NAVTRAEQUIPCEN 77-C-0005-1 


Struc- Decision 
ture Page Task No. Type Keyword - Task Area 
68 2.2.2.7.4.1.4.1] Type 3. Identify Survival Hold Release 
70 2.2.2.12.1 ZType 2 List all info. SAR Mission 
pertaining 
70 2.2.2.12.2 LType 2 List all info. SAR Mission 
pertaining 
70 2.2.2.12.3 rType 2 List all info. SAR Mission 
pertaining 
72 2.2.3.1.1 tType 2 List the info. SAR Mission 
- 75 2.2.4.8 Type 3. Identify Vessel Classification 
} 80 2.2.4.8.1 Type 3. Identify 
80 2.2.4.8.1.1 rType 2 List the critical... 
75 2.2.4.8 Type 3. Identify | Vessel Classification 
80 2.2.4.8.2 Type 3. Identify 
80 2.2.4.8.2.1 Type 3. Identify 
80 2.2.4.8.2.1.1 rType 2 List the critical... 
- 75 2.2.4.8 Type 3. Identify Vessel Classification 
80 2.2.4.8.2 Type 3. Identify 
80 2.2.4.8.2.2 Type 3. Identify 
80 2.2.4.8.2.2.1 ZType 2 State the critical... 
r 75 2.2.4.8 Type 3. Identify Vessel Classification 
80 2.2.4.8.3 Type 3. Identify 
b 80 2.2.4.8.3.1 Type 3. Identify 
80 2.2.4.8.3.1.1 LType 2 List the critical... 
75 2.2.4.8 Type 3. Identify Vessel: Classification 
80 2.2.4.8.3 Type 3. Identify 
80 2.2.4.8.3.2 Type 3. Identify 
> 80 2.2.4.8.3.2.1 cType 2 List the critical... 
- 676 2.2.4.6.1 Type 3. Identify Visual Rig 
b 77 2.2.4.6.1.1 Type 3. Identify 
77 2.2.4.6.1.1.1 Type 2 State rules for 
deter. 
76 2.2.4,6.1 Type 3. Identify Visual Rig 
77 2.2.4.6.1.2 Type 3. Identify 
77 2.2.4.6.1.2.1 Type 2 Identify 
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Struc- Decision 
ture Page Task No. Type Keyword Task Area 
76 2.2.4.6.1 Type 3 Identify Visual Rig 
77 2.2.4.6.1.2 Type 3. Identify 
77 2.2.4.6.1.2.2 Type 3. Identify 
76 2.2.4.6.] Type 3. Identify Visual Rig 
b 77 2.2.4.6.1.2 Type 3. Identify 
77 2.2.4.6.1.2.3 Type 2. Identify 
76 2.2.4.6.1 Type 3. Identify Visual Rig 
b 77 2.2.4.6.1.2 Type 3. Identify 
77 2.2.4.6.1.2.4 Type 3. Identify 
76 2.2.4.6.1 Type 3. Identify Visual Rig 
b 77 2.2.4.6.1.3 Type 3. Identify 
77 2.2.4.6.1.3.1 Type 3 Identify 
76 2.2.4.6.1 Type 3. Identify 
- 77 2.2.4.6.1.4 Type 3. Identify 
Pp 77 © 2.2.4.6.1.4.2 Type 2. Identify 
> 77 2.2.4.6.1.4.2.1 tType 2 State the critical... 
76 2.2.4.6.1 Type 3. Identify Visual Rig 
> 77 2.2.4.6.1.4 Type 3. Identify 
77 2.2.4.6.1.4.3 Type 2 Identify 
77 2.2.4.6.1.4.3 Type 2 State rules for 
identifying 
: 76 2.2.4.6.1 Type 3. Identify Visual Rig 
U 77 2.2.4.6.1.4 Type 3 Identify 
77 2.2.4.6.1.4.4 Type 2. Identify 
76 2.2.4.6.1 Type 3 Identify Visual Rig 
b 77 2.2.4.6.1.4 Type 3. Identify 
77 2.2.4.6.1.4.5 Type 3. Identify 
C 76 2.2.4.6.] Type 3. Identify Visual Rig 
77 2.2.4.6.1.5 Type 2 Identify 
C 76 2.2.4.6.1 Type 3. Identify Visual Rig 
> 78 2.2.4.6.1.6 Type 2. Identify 
82 2.2.5.1.2.1 tType 2 List the info. Cargo Mission 
pertaining... 
82 ee dee GY A ZType 2 List the info. Cargo Mission 
pertaining... 
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82 
89 
89 
89 
89 
90 
93 
93 
93 


93 


93 


Task No. 
2.5.1.2.3 


2.5.1.2.4 


2.11251 


2.7.1.2.2 


2.7.1.2.3 


2.7.1.2.4 


ie. PeDSe 


.3s 1.2.2 


3.1.3.2 
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Decision 


Type 
rType 


sType 
Type 
rType 
rType 


ZrType 


2 


2 


2 


Type 2 


Type 2 


xType 


3 


Type 2 


Type 2 


Type 2 
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Keyword 


List the Info. 
pertaining... 


List the Info. 
pertaining... 


List all .. 
pertaining... 
List...needed by... 
List info... that 


must... 


List info... that 
must... 

Determine if 
Determine whether 
State 

Determine 


Determine 


Determine if 


Task Area 


Cargo Mission 
Cargo Mission 
Passenger Transfer 
Mission 


Passenger Transfer 
Mission 


Passenger Transfer 
Mission 


Passenger Transfer 
Mission 


Flight Equipment 
Inspection 


Ordnance System 
Inspection 


Ordnance System 
Inspection 


Ordnance System 
Inspection 


Ordnance System 
Inspection 


Ordnance System 
Inspection 


Task No. 
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Analysis 


Task Name 


Fuel System Mal- 
function Detection 


Boost Pump 
Malfunction 


Transfer Pump 
Malfunction 


Compressor 
Malfunction 


Bypass Filter 
Malfunction 


Emergency Pump 
Malfunction 


External Engine Fire 
Internal Engine Fire 


Internal Aircraft 
Fire 


0i1 System Malfunc- 
tion Detection 


Gear Box Oil 
Malfunction 


Engine Oi] 
Malfunction 


Intermediate Gear 
Box Oil Malfunction 


Combining Gear Box 
Oil Malfunction 


Speed Decreaser 


Hydraulic System Mal- 
function Detection 


Hydraulic System 
Malfunction 


ASE Malfunction 
Detection 


APPENDIX C 


PILOT/ATO (TS) 


of PILOT/ATO Tape Slides 


Slide 


No 
F-] 


Fale 


F-1. 


F-1.5. 


F-1. 


F-1. 


F-1. 


. 


5. 


«e™ 


.5.2* 


See notes which follow this appendix 
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Old 


SF-1.7. 


SF-1.7, 


SF-1.7. 
SF-1.7. 


] 


11 


NA/ND Hidden/ 
Slide Na DM/DE Obvious 


NA 


DE 


ND 


ND 


ND 


NA 


obvious 


obvious 


hidden 


hidden 


Type 


Type 2 


Type 2 


Type 2 


Keyword 


identify 


state 


state 
whether 
or 
identify 
identify 


identify 


identify 


identify 


state 


identify 


identify 


identify 


Task No. 


1.1. 


— 
. 
— 


_ 
. 
— 


~ 
. 
_ 


4.3.1.1.] 


4.3.9.1.1 


~1.4.3.15.1.1 


-4.3.11.1.1.1 


-4.3.11.4.1.1 


-4.3.11.3.1.1 
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Task Name 


ASE Malfunction 


Landing Gear Failure 


to Lower 


Tire Blowout 


Postflight Inspection 


Droop Stops Failure 


Engine Malfunction 
Detection 


Engine Flameout 


Throttle-Caused 
Power Oscillation 


Partial Power Loss 
Compressor Stal] 


Electrically-Caused 
Overspeed 


Loss of Nf Signal 


Flight Control Mal- 
function Detection 


Loss of Tail Rotor 
Thrust 


Loss of Tail Rotor 
Control 


Elec'l System Mal- 
function Detection 


Generator 


Malfunction 


Converter 
Malfunction 


Total Electrical 
Failure 


26-Volt AC Trans- 
former Failure 


Slide 


No 
F-3 


F-3 


3.2 
3.3 


2320% 
.3.6* 


F-7.3. 


F-7.3. 


F-7. 


F-7. 


See notes which follow this appendix 
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SF-2.3.3 
SF-2.3.5 
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PF-3.1.1 
SF-4.3.2 


SF-4.3. 
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SF-4.3. 


Ww 
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SF-4.3.6 
SF-4.3.7 
SF-4.3.8 
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SF-5.3.7 


NA/ND Hidden/ 
DM/DE Obvious 


DE 
DE 


ND 
ND 
DE 
NA 


DE 
DE 


DE 


NA 


DE 


DE 


DE 


DE 


hidden 


hidden 


hidden 


Type 


Type 2 


Type 2 


Type 2 


Keyword 
identify 


identify 
state 
name 


identify 


identify 


identify 


identify 
identify 


identify 


identify 


identify 


identify 


identify 
identify 
identify 


identify 


] 


] 


] 
] 


] 


1 
1 


] 
] 
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Task No. 
.1.4.3.3.1.1 


21.12.1.3.4.2.1 


-1.4.3.8.1.1 
-1.4.3.7.1.1 


-1.10.2.3.1 
01.4.3.4.1.1 


-4.6.1.3.3.1.1 


-4.6.1.3.1.1 
-4.6.1.3.2.1.1 
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Task Name 


Torque Gauge 
Malfunction 


Shutdown with Total 
Electrical Failure 


Dual Engine Failure 


Single Engine 
Failure 


Electrical Fire 
Detection 


Determining Restart 
Feasibility 


Decaying Nr/Nf 
Settling with Power 


Power Settling 
Blade Stall 


.1.12.1.3.4.1.2 Locked Tailwheel 


Slide 
No. 


F-7.3.6 


F-7.2.4 


F-7.1.2* 


F-9.2.1 


F-9.2.2 


F-9.2.3 
F-11.1.1* 
F-11.1.2 


* See notes which follow this appendix 
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Old 


SF-5.3.10 


SF-5.3.11 


SF-5.2.1 
SF-5.2.2 


SF-5.2.6 


SF-5.1.2 


NA/ND Hidden/ 
Slide No. DM/DE Obvious 


DE 


ND 


ND 
DE 


DE 
ND 


obvious 


Type 


Type 2 


Keyword 
identify 


state 


state 
identify 
identify 
decide 
whether 
identify 
define, 
name, 
state 
identify 
identify 


state 


F-1.4, 


F-1.4. 


F-5, 


F-5.3. 


F-5.3. 
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PILOT/ATO (TS) 


Notes 


Measure of criticality of a mission is considered by the pilot. 
This information is used to decide whether to continue the 
mission or to abort. 


Identify cues and alerts for cabin fire by fumes, visible smoke, 
and/or fire. 


The cues and symptoms are given and the pilot must decide 
whether or not there is a gearbox malfunction. Phrases such as 
"the first indication of malfunction in the oi] system will 

robably be the transmission oi] pressure caution light" are 
used. Ks it is indicated in the tape slide, this is a "no ask 
from ground" procedure. 


Possible symptoms for a combining gearbox oi] malfunction are 
discussed. The pilot must identify the malfunction. Phrases 
such as “a complete break in the oil line will probably cause 
these indications (1) the combining gear box caution Tight will 
illuminate, (2)..." are used. 


Phrases such as "the hydraulic pressure caution light in the 
caution light panel will probably be your first indication of a 
malfunctioning, but remember a caution light could be accom- 
panied by a sudden burst of flight in the event of a massive 
break in the line" are used. 


The reaction speed is critical. The previously made decision is 
communicated to the pilot for execution via manuals and gauges. 


Determine whether it is throttle-caused oscillation or engine- 
caused oscillation. The procedure to test and determine is 
given. 


Symptoms are given but the procedure to obtain a diagnosis 
through the use of those symptoms is not mentioned. 


Phrases such as "on the other hand, if the aircraft is on the 
descent or on deck, faulty electrical input to the fuel control 
will probably result in rotor overspeed. The symptoms for 
electrically caused rotor overspeed are: (1) NFNR for ..., 
(2)..." are used. 
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Slide No. 


F-5.2.2 


F-5. 


fo 
w 


F-7.1.2 


F-7.11.1.1 
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Notes 


As symptoms, phrases such as _"... depending upon Flight 
condition, aerodynamic coupling may produce right ro and 

pitch down. The rate of yaw will depend upon the amount of 
torque applied to the ..." are used. It may be critical 

because it is said that "loss of tail rotor thrust demands 
immediate action". It may cause total loss of control which, 

in turn, may cause landing on the nose. Also there are phrases 
such as "(1) begin a 70 Ng minimum power descent or autorotation 
depending upon the intensity of the vibration". 


Some alternatives are given for yaw controlling the aircraft 
with loss of tail rotor control, but the criteria for choosing 
any one of them is not mentioned. 


The pilot must figure out how much gas is left without the use 
of working instruments (due to lack of generator). 


Same as F-7.3.2 


Phrases are used such as "the first indication of engine failure 
will probably be the sound of the engine unwinding. This will 
be followed by ...". 


Whether or not to attempt a restart is investigated. Phrases 

are used such as "the ultimate decision as to whether or not a 
restart should be attempted rests with the pilot in command. 

In making this decision he must rely on his knowledge of NATOPS, 
his good judgment and his personal experience. There are no 
hard and fast rules of attempting to restart after an engine is 
flamed or has been secured in flight. But there are some 

general guidelines which can be followed. An engine restart 
should be attempted under the following conditions: (1)...-; 
(2)..., (3)... Remember, do not attempt to restart if the engine 


is in flame because of an internal or external fire." 


Critical. Requires rapid identification and correction action. 


14] 
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NA/ND 


ND 
ND 
ND 


APPENDIX D 
CREW (TS) 
Analysis of Crewman Tape Slides 
Task Name rere 
Planned Ditching CRF-1.3.4 
Search & Rescue Equipment CRF-1.5.5/6 
Crewman Safety Belt CRF-1.6.3 
Aircraft Handling #1 PF-3.2.1 
Aircraft Handling #2 PF-3.2.2 
Aircraft Movement Safety Rules PF-3.2.4 
Signature Cards PF-2.1.3 
Immediate Ditching Procedure CRF-1.3.3 
Bailout Procedure CRF-1.3.5 
Total Electrical Failure CRF-1.3.7 
Internal & External Engine Fire CRF-1.3.9 
Hand Pressure Signals CRH-1.3.3 
Cabin and/or Electrical Fire CRF-1.3.11 
~ Hung Droop Stop CRF-1.3.14 
Cargo Transfer Safety Precaution CRH-1.4.1 
Cargo Rigging Safety Precaution CRH-1.4.5 
Normal Hoisting Operations CRH-1.3.7 
Connect/Disconnect Refueling Hose -CRH-2.1.4 
Passenger Survival Gear CRH-2.1.4 
Swimmer's Equipment CRW-1.2.1 
Required Flight Gear CRF-1.6.7 
Regaining Lost UHF Communication SE-4.9.2 
LAMPS Frequency Circuits SE-4.9.3 
Aircraft Positioning During HIFR CRH-2.1.2 


Keyword 


state 

name, state 
describe, name 
state 

state 

state 
demonstrate 
state 

state 
identify 
state 

state 

state 

state 

state 

state 

state 

state 

name 
identify, state 
name 

name 

name 


state 
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Task Name 


Sonobuoy Loading Safety 
Inspecting JAU-1B CAD 


Inspecting Sonobuoy Launch 
Container 


Major Preflight Inspection Areas 


Radar Interpretation - 
Land Mass 


Radar Interpretation - 
Heavy Weather Contact 


Radar Interpretation - 
Surface Contact 


Radar Interpretation - 
Airborne Contacts 


Sound Velocity 


Old 
Slide No. 


OR-2.1.8 
OR-1.1.5 


OR-1.1.3 


PF-3A.2.7 
NA-2.1.1* 


NA-2.1.2* 


NA-2.1.3* 


NA-2.1.4* 


AC-1.2.11 


*See notes which follow this appendix. 
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NA/ND 


ND 
ND 
ND 


ND 
ND 


ND 


ND 


ND 


ND 


Keyword 


state 
state 


name 


name 


name 


name 


name 


name 


describe 


Slide No. 


NA-2.1.1 


NA-2.1.2 


NA-2.1.3 


NA-2.1.4 
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CREW (TS) 
Notes 


This slide prepares the crewmen for distinguishing between a 
target and a land mass which itself is a classification and, 
therefore, a decision task. Only the attributes (features of a 
land mass on radar (one of the two classes) is listed and the 
crewman is supposed to classify the object as a land mass if 

he identifies those attributes. 


"Name three radar display characteristics of heavy weather" is 
the objective of this course. These characteristics (features) 
are listed. 


The three radar display characteristics of surface contact are 
listed. 


The three radar display characteristics of an airborne contact 
are listed. One of them (fast movement) is mentioned to be the 
distinguishing characteristics between airborne contact and 
surface contact. 


144 


. . . . . . . 
> > > > > > > 


> 
w 
. 

~ 
ine) 


> 
w 
— 
ine) 


-10.1.6. 


-1.12.1.3. 


fo) 


> > 
. . 


NAVTRAEQUIPCEN 77-C-0005-1 


APPENDIX E 
PILOT/ATO (WB) 


Analysis of PILOT/ATO Workbooks 


Section 
Task Name No. 
Transmission System F-1.8.1 
" F-1.8.2 
- F-1.8.3 
" F-1.8.4 
F-1.8.5 
" s F-1.8.6 
Fire Detection and F-1.7.1 
Extinguishing System 
" F-1.7.2 
: F-1.7.3 
: F-1.7.4 
" F-1.7.5 
" F-1.7.6 
: F-1.7.7 
Fuel System F-1.6.1 
Operation 
2 F-1.6.2 
" F-1.6.3 
x F-1.6.4 
i F-1.6.5 
‘ F-1.6.6 
” F-1.6.7 
Fuel Control F-1.5.9 
Divider Setting 
Flight Control F-1.3.1 
System 
Disengage Rotor F-1.2.7 
Envelope 
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Old Sec- 
tion No. 


SF-1.10.1 


SF-1.9.1 


SF-1.8.1 


SF-1.7.10 


SF-1.4.1 


SF-1.3.7 


NA/ND Hidden/ 
DM/DE Obvious 


ND 
ND 
ND 
ND 
ND 
ND 
ND 


ND 
ND 
ND 
ND 
ND 
ND 
ND 


ND 
ND 
ND 
ND 
ND 
ND 
ND 


ND 


DE 


Keyword 


describe 


state 


describe 


determine 
if 


Task No. 


. 


NAVTRAEQUIPCEN 77-C-0005-1 


Task Name 
Airspeed 
Density 
Torque 
Endurance 
Gross Weight 


Single Engine 
Airspeed 


VMC Recovery to Land 
Land Recovery 

Normal Take-Off 

Taxi Procedure 


Airframe System 


Rotor System 
Operation 


Rotor System 
Operation 


Wheel Brakes System 


Section Old Sec- 
No. tion No. 

F-1.2.6 SF-1.3.6 

F-1.2.5 SF-1.3.5 

F-1.2.4 SF-1.3.4 

F-1.2.3* SF-1.3.3 

F-1.2.2 SF-1.3.2 

F-1.2.1 SF-1.3.1 

F-1.1.4*  SF-1.1.4 

F-1.1.3 SF-1.1.3 

F-1.1.2 SF-1.1.2 

F-1.1.1 SF-1.1.1 

F-3.11.1 SF-2.111 

F-3.11.2 

F-3.11.3 

F-3.11.4 

F-3.11.5 

F-3.11.6 

F-3.10.1  SF-2.10.1 

F-3.10.2 

F-3.10.3 

F-3.10.4 

F-3.10.5 

F-3.10.6 

F-3.9.1 SF-3.9.1 

F-3.9.2 


See notes which follow this appendix 
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NA/ND Hidden/ 
DM/DE Obvious 


DE 
DE 
DE 
DE 
DE 
DE 


ND 
ND 
ND 
ND 
ND 
ND 
ND 
ND 
ND 
ND 
ND 


Type 


Keyword 

calculate 
determine 
determine 
determine 
calculate 


determine 


state 
state 
state 
state 


describe 


describe 


describe 


describe 


. 


a 


. 
~ 


. 
— 


Task No. 


10. 


10.1. 


10. 


10. 


. 


‘lis 


Dn DD DH 
> FSF SP LFS LP 


4.3. 1111.1.2 


-4.3.1.1.1.1.3 


. 


4.3.1.1.1.4 


-1.4.3.1.1.1.1.2 


64.3. 00.1.11 


4.3. 0001.11 


3.2.4 


NAVTRAEQUIPCEN 77-C-0005-1 


Task Name 


Wheel Brakes System 


Blade Track System 


Hydraulic System 


ASE System 


Battery Start 


Section 
No. 


F-3. 
F-3. 
F-3. 
F-3. 


no F F F F F FD AHARHARAR HD @w w0©woewwwWO WwW WO wW wo wo 


on 
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Old Sec- 
tion No. 


SF-2.8.1 


SF-2.6.1 


SF-2.4.1 


SF-2.2.1 


NA/ND 
DM/DE 


ND 


Hidden/ 
Obvious Type 


Keyword 


describe 


describe 


describe 


— 
. 


a | 


Task No. 


ine) 

Oo Ww W Ww WwW Ww 
— 
w 


-1.10.1.1.8.1.1 


.10.1.1.8.1 
.10.1.1.8.2 
.10.1.1.8.4 
.10.1.1.8.3 | 
.10.1.1.8.1.2 


-1.70.1.1.8.1.3 


Task Name 


NAVTRAEQUIPCEN 77-C-0005-1 


Section 
No. 


Precision Approach F-3.1.1 


Normal Approach F-3.1.2 
Rustlick F-4.2.3 
Vids Fill-Out F-4.2.2 
Yellow Sheet F-4.2.] 


Navigation System F-5, 


Pitot Airspeed 


System 


Power Plant System F-5.4.7 
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Old Sec- 
tion No. 


SF-2.1.3 
SF-2.1.4 
PF-3A.1. 3 
PF-3A.1.2 
PF-3A.1.1 


SF-4.6.1 


SF-4.7.1 


SF-4.5.1 


NA/ND Hidden/ 
DM/DE Obvious 


ND 
ND 
ND 
ND 


Type 


Keyword 
state 
state 
state 
Fill out 
list 


describe 


NAVTRAEQUIPCEN 77-C-0005-1 


Section Old Sec- NA/ND Hidden/ 


Task No. Task Name No. tion No. DM/DE Obvious Type Keyword 
1.1.11.2.3.2.1 Recovery w/ loss of F-5.1.5*  SF-7.1.2 DE state 
Tail Rotor Control ; 
1.1.10.2.2.2.1 Running Autorotation F-5.1.3 SF-4.1.3 ND state 
Landing ; : 
1.1.10.2.2.1.1 Fuel Flare Auto- F-5.1.2  SF-4.1.2 ND state 
rotation Landing 
1.1.10.2.1.4 Emergency RPM Con- F-5.1.1*  SF-4.1.1 DE state 
trol Approach 
Communications F-7.7.6 ND describe 
System 
1.8.2.5.5.1 ” F-7.7.5 " iw 
1.8.2.5.5.2 2 F-7.7.4 x . 
1.8.2.5.5.3 " F-7.7.3 u u 
1.8.2.5.5.1.2 z F-7.7.2 " " 
1.8.2.5.5.1.3 w F-7.7.1 SF-5.6.1 : " 
Caution Lights F-7.6.6 ND describe 
System 
1.8.2.5.5.1 : F-7.6.5 " a 
1.8.2.5.5.2 i F-7.6.4 " " 
1.8.2.5.5.3 " F-7.6.3 ! " 
1.8.2.5.5.1.2 i F-7.6.2 " ‘ 

: F-7.6.1 SF-4.16.1 " 4 
1.8.2.2.1 Lighting System F-7.5.5 ND describe 
1.8.2.2.4 t F-7.5.4 : " 
1.8.2.2.5 " F-7.5.3 : i 
1.8.2.2.2 " F-7.5.2 “ " 
1.8.2.2.3 " F-7.5.1 SF-5.5.1 - m 


See notes which follow this appendix 
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-NAVTRAEQUIPCEN 77-C-0005-1 


Section Old Sec- NA/ND Hidden/ 


Task No. Task Name No. tion No. DM/DE Obvious Type. Keyword 
.1.4.3.11.5.1.1 Electrical System F-7.4.7 ND . describe 
-1.4.3.11.5.1 " F-7.4.6 " e 
£1.4.3.11.5.2 " F-7.4.5 " i 

1.1,4.3.11.5.4 * F-7.4.4 " " 

.1.4.3.11.5.3 " F-7.4.3 " ” 

-1.4.3.11.5.1.2 : F-7.4.2 " " 

-1.4.3.11.5.1.3 F-7.4.1 SF-5.4.1 " ii 

-1.10.2.4.1 Ditching Station F-7.2.8* SF-5.2.11 DE state 

-1.10.2.4 Immediate Ditching F-7.2.7* SF-5.2.10 DM hidden Type 2. state 

.1.10.2.4.1 Controlled Ditching F-7.2.6 SF-5.2.9 ND state 

-1.10.2.6 Water Takeoff F-7.2.5 SF-5.2.7 ND state 

.1.4.3.10.5 Internal A/C Fire F-7.2.3 SF-5.2.3 ND state 
Likelihood 

.1.11.1.3.2.3.3 Single Engine F-7.1.6 SF-5.1.6 ND state 
Recovery 

.1.11.1.3.2.3.1 Jettisoning F-7.1.3 SF-5.1.3 ND state 

1.1.10.2.3.2 Secured Engine F-7.1.1 SF-5.1.1 ND state 

Restart 

.8.2.4.2.1.1 Weapons Systems F-9.8.7 NA 

.8.2.4.2.1 i. F-9.8.6 " 

.8.2.4.2.2 iu F-9.8.5 " 

-8.2.4.2.4 ‘ F-9.8.4 s 

.8.2.4.2.3 " F-9.8.3 " 

.8.2.4.2.1.2 7 F-9.8.2 . 

.8.2.4.2.1.3 Weapons System F-9.8.1 NA 

842.452. 1e1 Electrical Release -F-9.7.7 ND describe 
System 


See notes which follow this appendix 
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NAVTRAEQUIPCEN 77-C-0005-1 


Section Old Sec- NA/ND Hidden/ 


Task No. Task Name No. tion No. DM/DE Obvious Type Keyword 
1.8.2.4.2.1 Electrical Release F-9.7.6 ND describe 
System 
1.8.2.4.2.1 : F-9.7.5 " : 
1.8.2.4.2.4 " F-9.7.4 . " jc 
1.8.2.4.2.3 " F-9.7.3 " u 
1.8.2.4.2.1.2 7 F-9.7.2 " " 
1.8.2.4.2.1.3 " F-9.7.1 SF-7.10.1 " . 
Sonobuoy System F-9.6.3 ND describe 

; F-9.6.2 i " 

" F-9.6.1 SF-5.9.1 " " 
1.8.2.4.8.1.1 ALR-54 System F-9.5.7 ND describe 
1.8.2.4.8.1 " F-9.5.6 " “ 
1.8.2.4.8.2 F-9.5.5 " i 
1.8.2.4.8.4 7 F-9,5.4 " " 
1.8.2.4.8.3 ut F-9.5.3 " u 
1.8.2.4.8.1.2 0 F-9.5.2 " u 
1.8.2.4.8.1.3 ‘ F-9.5.1  SF-7.4.1 " " 
1.8.2.4.7.1.1 MAD System F-9.4.7 ND describe 
1.8.2.4.7.1 " F-9.4.6 " " 
1.8.2.4.7.2 F-9.4.5 ) u 0 
1.8.2.4.7.4 " F-9.4.4 " u 
1.8.2.4.7.3 : F-9,4.3 " " 
1.8.2.4.7.1.2 M F-9.4.2 " i 
1.8.2.4.7.1.3 " F-9.4.1 SF-7.3.1 . " 
1.1.6.1.3.6 MAD Use F-9.3.5*  SE-1.4.4 NA 


* 
See notes which follow this appendix 
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_ ~ oa . NAVTRAEQUIPCEN 77-C-0005-1 


Section — 
Task No. . Task Name No. 

MAD Pattern F-9.3.4 

Strengths 

MAD Localization F-9,.3.3* 

MAD Procedures F-9.3.1 
1.1.4.2.3.1 Max. Gross Weight F-9.1.1 

Takeoff 


LN66 HP Radar System F-11.5.3 


1.8.2.3.7.1 " F-11.5.2 
1.8.2.3.7.2 . F-11.5.1 
.1.8.4.2.1.1 Cargo Hook System F-11.4.6 
1.8.4.2.1 F-11.4.5 
1.8.4.3 ” F-11.4.4 
1.8.4.2.2 " F-11.4.3 
1.8.4.2.1.2 F-11.4.2 
1.8.4.2.1.3 i F-11.4.1 
1.8.5.3.1.1 Environmental System F-11.3.7 
1.8.5.3.1 " F-11.3.6 
1.8.5.3.2 " F-11.3.5 
1.8.5.3.4 " F-11.3.4 
1.8.5.3.3 i. F-11.3.3 
1.8,$23:12 i F-11.3.2 
1.8.5.3.1.3 . F-11.3.1 
Radar Offset F-11.2.2* 


Approach 


‘i See notes which follow this appendix 
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Old Sec- 
tion No. 


SE-1.2.8 


SE-1.2.7 


/SE=1.2.5 
Seas 


SF-4.12.1 


SF-8.4.1 


SF-8.3.1 
SE-4.8.3 


NA/ND Hidden/ 
DM/DE Obvious 


NA 


NA 
ND 
ND 


ND 


Type 


Keyword 


state 


state 


describe 


describe 


DM obvious Type 2 state 


Section No. 


F-1.2.3 


F-1.1.4 


F-7.2.8 


F-9.3.5 


F-11.2.2 


NAVTRAEQUIPCEN 77-C-0005-1 
PILOT/ATO (WB) 


Notes 


There is a hidden decision making task inside the whole task 
"determine the required fuel reserve"; however, the procedure 
for its calculation is given "10% of fuel on board or 20 minutes 
of flight time, whichever is longer". 


There are two hidden aecision making tasks inside the whole 
task: 

(1) "Determine destination weather", and- 

(2) “Determine traffic pattern", 
neither of which possesses a procedure. However, both are said 
to be “asked from the tower". Therefore, they are non-decision 
tasks with respect to the pilot. 


The procedure is given in the form of "If (condition), then 
action)", which is the translation of a table look up. This 
1s a means of communicating a decision previously made to a 
decision executer. 


There is a hidden decision “determine type of landing to be 
made"; however, the procedure to perform this determination is 
also given. 


This task can be considered as a decision execution task. Since 
the decision as to "should an immediate ditching, planned 
ditching, or none be performed" has previously been made for 

all possible states of nature, the result of this decision is 
now being communicated to the pilot for execution. 


Item B-2 of the procedure says, "Either secure engine and rotors 
(release harness and abandon aircraft) or follow single engine 
water takeoff procedure". However, it has not been Suggested 
how to decide between these two alternatives. Therefore, this 
remains as a decision task to be performed by the pilot. 


This task is potentially a very good technical and also critical 
(in terms of the objective of the mission) decision task. The 
course materials are not found in the workbook pack. (It might 
be classified. ) 


It is a potential decision task; however, the corresponding course 
material is not found in the workbook pack. (It might be 
classified. ) 


The item 2 of the procedure: "Determine if a direct or offset 
approach will be used" is a decision task. In the description of 
this item, it is said that "The ATO will decide if a direct or 
offset approach will be used". No completely specified criteria 
for this selection has been given. 
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Fuel Starvation (6) 
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APPENDIX G 
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APPENDIX H | 
SCENARIOS FOR PROTOCOL ANALYSIS 


Consider yourself engaged in each of the following scenarios. 
Describe what might happen in terms of the possible alternative courses 
of action, possible states of nature, and the outcomes associated with 
each combination of a state of nature and a course of action. 


A. ~ Engine Quits 


(1) You have a single engine failure while flying over land. 
One engine is sufficient for the weight of the aircraft 
and there is a safe landing site in the area. 


(2) You have a single engine failure while flying over land. 
One engine is sufficient for the weight of the aircraft 
and there is an unsafe landing site in the area. 


(3) You have a single engine failure while flying over land. 
One engine is not sufficient for the weight of the 
aircraft and there is a safe landing site in the area. 


(4) You have a single engine failure while flying over land. 
One engine is not sufficient for the weight of the 
aircraft and there is an unsafe landing site in the 
area. 


(5) You have a dual engine failure while flying over land. 


(6) You have a fuel starvation while flying over land and 
you don't have enough altitude to bail out. 


(7) You have fuel starvation while flying over land and you 


have enough altitude to bail out. However, the site 
is dangerous. 
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You have fuel starvation while flying over land and you 
have enough altitude to bail out. The site is safe. 


You have a single engine failure while flying over water. 
One engine is suffucient for the weight of the aircraft. 


You have a single engine failure while flying over water. 
One engine is not sufficient for the weight of the aircraft. 


You have a dual engine failure while flying over water. 


You have fuel starvation while flying over water. 


or Wave-Off 


You have attempted landing. The aircraft carries a heavy 
cargo and the environmental condition is good. The landing 
site is prepared. 


You have attempted landing. The aircraft carries a heavy 
cargo and the environmental condition is good, but the 
landing site is unprepared. 


You have attempted landing. The aircraft carries a heavy 
cargo and the environmental condition is bad, but the 
landing site is prepared. 


You have attempted landing. The aircraft carries a heavy 


cargo and the environmental condition is bad. . There is 
an unprepared landing site. 


175 


NAVTRAEQUIPCEN 77-C-0005-1 


(5) You have attempted landing. The aircraft carries a light 
cargo and the environmental condition is good. The landing 
site is prepared. 

(6) You have attempted landing. The aircraft carries a light 
cargo and the environmental condition is good, but the | 
landing site is unprepared. 

(7) You have attempted landing. The aircraft carries a light 
cargo and the environmental condition is bad, but the 
landing site is prepared. 

(8) You have attempted landing. The aircraft carries a light 
cargo and the environmental condition is bad. There is an 
unprepared landing site. 

Ditching 

(1) There is an alert which suggests an emergency ditching, 
but you are over the land. 

(2) There is an alert which suggests an emergency ditching 
and you are over the water. 

(3) There is an alert which suggests a planned ditching, but 
you are over the land. 

(4) There is an alert which suggests a planned ditching and 


you are over the water. 
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In the protocol analysis performed, based on the above scenarios, 
the answer to the following questions were pursued. 


Single Engine Failure - Over Land 


1. Do you always know whether one engine will be sufficient to continue 
on the mission or not? 


2. How do you know? (or) Why don't you know? 


3. If one engine is sufficient, do you still look for a safe landing 
site? 


4. Does the importance of the mission enter into your decision on what 
to do? 


5. At what point do you decide whether to try to land or to continue? 


6. If you find a safe landing site, what makes you decide whether to 
land or to continue? 


7. If you continue, do you tend to make changes in the objectives of 
the mission? That is, if the mission has more than one purpose, 


do you try to accomplish some but not others? 


8. If you continue, do you evaluate your chances of being successful? 
How? 


9. If you can't find a safe landing site, do you continue looking for 
one or do you try something else? 
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If one engine is not sufficient to continue and you find a safe 
landing site, do you attempt to jettison or to land? 


If one engine is not sufficient, what is usually the reason? 
Load? 


(If Yes) In that case, would you always try to jettison first, 
and then decide whether one engine is sufficient to continue? 


In general, what makes you decide whether to jettison or not? 
Does the safety of the landing site make a difference? 


How do you decide whether it is better to land and abort the mission, 
or to jettison and continue? 


Can you generally place a monetary value on the loss incurred by 
jettisoning? 


Is it generally the same each time? (i.e., you tend to get rid 
of the same things?) 


(Depending on the cost involved in jettisoning) would you try to 
land in an unsafe landing site rather than to jettison? 


Of different possibilities with single engine failure, one engine 
not sufficient, and unsafe landing site; 


(a) Which will you try first? Why? 
(b) What does it depend on? 


(c) Are the environmental conditions more important or the possible 
losses that you might incur? 
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20. At what point do you start thinking about bailing out? 

21. You have attempted to jettison, one engine is still not sufficient, 
and you can't find a safe landing site, which will you do first: 
(1) try to land anyway, (2) bail out, or (3) attempt restart? 
What does it depend on? . 


22. (Same question following "attempt restart" and failure: land or 
bail out?) 


Dual Engine Failure - Over Land 

Discuss the possibilities as above: 

What do you do first? Why? Are there other possibilities? 

Do you always do the same thing first? What does it depend on? 


If you have a small chance of landing safely by auto-rotating, do you 
take it? (rather than lose the helicopter?). 


Over Water 


Questions concerning the importance of the mission and the losses involved 
in jettisoning apply here. 


Land or Wave-Off 
1. What is your first consideration when deciding whether to land or 


wave off? Could you rank the following in terms of their importance 
in your decision? 


179 


NAVTRAEQUIPCEN 77-C-0005 -1 


(a) Weight of cargo 

(b) Environmental conditions 

(c) Landing site (prepared/unprepared) 
(d) Speed of helicopter 

(e) Sink rate | 

(f) Fuel amount 


Are there other considerations? 


How do you decide whether a particular situation is safe or not? 
Could you look at all the outcomes and rate them in terms of the 
dangers involved? For example, you have a heavy cargo, the 
environmental conditions are good, and you have decided on a 
running landing, but you are going too fast, would you try to 
land, or would you wave off? If you try to land (because of 

low fuel) is it safer (or less dangerous) than if you tried to 
land under the same conditions but with a high sink rate (rather 
than with high speed)? 


(a) If you have plenty of fuel, would you always wave-off on 
the first approach: 


If you are going too fast? 
If you have too high a sink rate? 


(b) Does each wave-off "cost" something? That is, aside from 
using up fuel, are there other dangers involved in waving 
off? 


How would you rate these dangers? 
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(c) Are your decisions different (re-land or wave-off) depending - 
on whether you are planning a hover - or a running - 
landing? 
How? Why? 


Ditching 


1. How many different types of alerts are there that suggest a ditching 
situation? 


2. How different are they from each other? 


3. Can you always tell whether the alert suggests an emergency versus 
a planned ditching? In general, how often does each happen? 


4. If the alert suggests a planned ditching, in general, how long do 
you have until you have to make a decision? 


5. Do you start looking for a landing site as soon as you become 
aware of the alert? 


6. If you are over land, do you try to land as soon as possible? 


7. If you are over water, how long do you wait until you decide to 
ditch? 


8. When you first become aware of the alert, is there a way to estimate 
how likely it is that it is a correct (or an incorrect) alert? 
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Do the “alert” cues change over time, so that, for example, you may 
first decide to look for a proper landing site (over land, during 
planned ditching), and then change your mind and decide to land 

as soon as possible? 


If so, what are the changes in the alert condition that produce this 
switch? Are they clear-cut changes or is it a matter of "judgment"? 


If it is a matter of judgment, can you elaborate? 


Do you consider what it will cost if you ditch? To you? To the 
Navy? 
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APPENDIX I 
TRIP TO LOS ANGELES POLICE ACADEMY 


The Los Angeles Police Academy in Pasadena has recently constructed 
a simulator for training decisions in "shoot/no shoot" situations. That 
is, the simulator focuses on the problem of when to shoot rather than 
how to shoot or at what target. The simulator is built in a one-room, 
two-story, cubic building. A giant movie screen fills one entire wall 
from floor to ceiling with movie projectors located in a small booth 
near the ceiling on the opposite wall. A quadraphonic sound system 
accompanies the films. 


The police cadet stands in front of the screen with a special 
laser pistol that resembles his service revolver. The filmed scenario 
places the cadet in a situation in which he must decide whether or not 
to shoot at a criminal who is threatening his life. Two video cameras 
monitor the cadet during the scenario. One is located near the movie 
projector and the other is mounted directly above the screen itself in 
order to provide a front view. 


As the cadet shoots, a light-sensing device located in the rear 
of the room pinpoints the location of the "shot" on the screen to an 
accuracy of one inch. This information, plus the exact movie frame that 
was showing at the time, are input to a Nova mini-computer and stored 
for later processing. At the end of the scenario, a videotape is prepared 
using "split screen" technique, of the critical portions of the exercise. 
The action on the screen is placed on the top half of the video image 
and the cadet is placed in the bottom half. This tape is used to 
review the cadet's actions at a later time. By using both the front 
and rear cameras, the location of the cadet and the location of his 
shot on the screen can be determined to an accuracy of one inch and 
within 1/24 of a second. 
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Although there is no formal training in either probability theory 
or decision theory, there are extensive courses in the established 
important attributes that should be considered when deciding whether or 
not to shoot in a given situation. Some of these attributes are: 


(1) legal aspects; 
(2) moral aspects; and 
(3) personal safety, safety of bystanders, safety of fellow officers. 


There is, thus, no relative weighting scheme for the attributes 
nor decision rules that translate the utilities of the attributes into a 
decision. There are, however, general guidelines that help the cadets 
make the decision. Since the shoot/no shoot decision must usually be 
made in less than one second, there is little or no time to consider 
attributes, utilities, or probabilities at that time. Consequently, the 
cadets are taught to prune the decision tree before the actual situation 
arises. The more aspects considered and dealt with early, the less to 
consider when the time comes to make the decision itself. 
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APPENDIX J 
DECISION TASK CLASSES 


SINGLE ATTRIBUTE DECISION MAKING 


General Definition: A decision where the action taken and the 
state of the world in combination determines the outcome (payoff) where 
the outcome (payoff) is a single value dimension (attribute). 


Amplification: A decision maker is frequently asked to choose 
between courses of action whose probable consequences are characterized 
by a single value attribute (for example, deciding on the number of 
sonobugys of a particular type to take on a mission). The outcomes, 
depending on the number taken and the number of submarine contacts 
made (state of nature) will determine the single attributed outcome: 
adequacy of sonobuoy type supply on hand. Adequacy of supply is a single 
outcome ranging from some numerical shortage to a numerical surplus. 


Rules: Identify the decision outcome for each action/state of 
the world. The outcome may be either estimated directly by experts, 
predicted by modeling, or predicted by experimental evidence. Each 
outcome may be either wholly subjective, subjective estimates on an 
objective scale, or objective measurements on an objective scale. 


Pitfalls and Limitations: The most important pitfall is the 
representation of an outcome by a single point estimate when the outcome 
itself may have a range or may be a probability density function. 


Interactions with Other Decision Elements. No major interactions. 


Prerequisites. There must be a method or mechanism that permits 
(trains or projects) outcome estimation. 
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MULTIPLE ATTRIBUTE DECISION MAKING 


General Definition: A decision situation where the action taken 
and the state of the world in combination determines the outcome (payoff) 
where the outcome (payoff) involves multiple value dimensions (attributes). 


Amplifications: Decision makers frequently choose between courses 
of action whose probable consequences are each characterized by multiple 
value attributes. For example, the decision of whether or not to ditch 
a LAMPS helicopter involves a number of value relevant factors such as 
worth of aircraft, safety to individuals in the helicopter, safety to 
those on the ground, adverse reaction if it is shown later by investigation 
that the ditch was unnecessary, and so on. In making a decision in such 
a situation, all such value-relevant factors must be identified for each 
alternative/state of nature. | 


Rules: Identify, for each action alternative/state of nature, 
the set of outcomes across all value dimensions. Outcomes may be either | 
estimated directly by experts, predicted by modeling, or predicted by 
experimental evidence. Each outcome may be either wholly subjective, 
subjective estimates on an objective scale, or objective measurement on 
an objective scale. 


Pitfalls and Limitations: The most important pitfall: is the 
representation of an outcome by a point estimate when in fact the outcome 
is a range or may itself be a probability density function. 


Interactions with Other Decision Elements: No major interactions. 


Prerequisites: There must be a method or mechanism for generating 
outcome estimates. This may involve training outcome estimation, providing 
standard lists of outcome estimation, and so on. . 
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WELL DEFINED DECISION MAKING SITUATIONS 


General Definition. A decision situation where the action 
alternatives, states of nature, and outcome set is completely describable, 


~well defined, well understood in advance of the actual decision making 
situation. 


Amplification. A key to any decision situation is problem 
structuring. The prerequisites for structuring are: (1) knowledge of 
the complete set of action alternatives to be considered, and (2) 
knowledge of all likely states of nature that will be involved in the 
situation. If these prerequisites are met, the situation is well defined 
and can easily be structured. For example, in the event of engine 


failure in the LAMPS helicopter, the craft may be over land or over water 
and the failure may be due to fuel starvation, dual engine failure, or 


single engine failure. The alternatives are clear if, for example, single 
engine failure occurs: the pilot may land, continue, abort the mission, 

or look for a safe site if the aircraft is over an unsafe landing site 
(state of nature). In this example, the decision situation is well defined. 


Rules: There are two possible mechanisms used to present the 
detailed structure of well-defined decision making situations: (1) the 
payoff (or loss) table, and (2) the decision tree. The general procedure 
is to express the decision alternatives and, for each alternative, list 
consequences under each state of nature. In one case, the result is in 
matrix form, in the other the result is in tree diagram form. If there 
are a sequence of decisions, a decision tree is usually best for 
structuring. 


Pitfalls and Limitations: A tree or matrix summarizing a decision 


situation forces closure in a specific decision situation. There is a 
tendency to view the decision in terms of the tree or matrix only, rather 
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than realizing that the actual decision may involve more than what is just 
shown in the structure. That is, the closure may be in error with a 
major element of the decision omitted. 


Interactions with Other Decision Elements: The outcomes to be 
displayed are either single or multiple and are generated as indicated 
earlier. Also, there may not be enough time to create a large structure. 


Prerequisites: A tree or matrix structure should be developed or 
provided. If developed, the basic material for constructing such a 
structure must be provided. The set of possible action alternatives 
should be provided with the outcomes for each prespecified alternative 
state of nature. The set of action alternatives should be mutually 
exclusive and exhaustive. 
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AMBIGUOUS DECISION MAKING SITUATIONS 


General Definition: A decision situation where the action 
alternatives, states of nature, and outcome set is neither well defined 
nor completely understood prior to or during the actual decision making 
situation. 


Amplification: The prerequisite for structuring any decision 
situation as indicated previously are (1) knowledge of the complete set 
of possible action alternatives to be considered in the decision 
situation and (2) knowledge of all likely states of nature that will 
be involved in the situation. In ambiguous situations, either one or 
both of these preconditions for well defined decision making structuring 
fail. There are two distinct kinds of failure: (1) the set of states 
of nature or the set of possible action alternatives is so large that 
for practical purposes it cannot be represented in matrices or tree 
diagrams, (2) the set of states of nature of the set of possible action 
alternatives cannot be anticipated completely in advance of the actual 
decision situation. For example, what should the pilot or ATO do when 
the LAMPS looses contact with a submarine. While the general set of 
alternatives and states of nature are known, the complete sets are not. 
In any given situation, there may be alternatives or states of nature 
unique to that situation. 


Rules: As in the case of well defined situations, structure 
what is known in matrices or decision trees and specify unknowns where 
appropriate. In addition, indicate that part of the structuring must 
occur at the time of the actual decision. The concept of closure must 
be firmly addressed as well as sufficiency of the alternative set. 
Sensitivity analysis should be conducted to determine decision sensitivity 
to structure. 
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Pitfalls and Limitations: In particular, any structure provided 
tends to take on more value than is warranted. Preliminary structuring may 
preclude additional structuring as an expedience. There are no complete 
rules for specifying when all the likely and relevant states of nature 
have been specified. 


Interactions with Other Decision Elements: The outcomes to be 
displayed are either single or multiple and are generated as indicated 
earlier. Also there may not be enough time to create a large structure. 


Prerequisites: A tree or matrix should be developed or provided. 
Particular attention should be given to providing methods for forcing 
consideration of additional states of nature and alternative combinations. 
The set of alternatives and states of nature provided, while not exhaustive, 
must be mutually exclusive. 
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TIME RELAXED DECISION MAKING 


General Definition: A decision situation where the decision maker 
has sufficient time to completely structure the decision in terms of number 
of attributes, alternatives, states of nature, and outcomes as well as 
selecting the best alternative. In short, there are no time constraints. 


Amplification. The decision making situations where a decision 
maker has literally "plenty" of time to structure and make a decision 
represent time relaxed decision making. For example, during pre-mission 
planning in LAMPS operations, there is often enough time for discussion 
and planning. | 


Rules: There are no specific rules except to be thorough in all 


aspects of decision structuring and alternative selection. Moreover, 
there is enough time to conduct sensitivity analyses on alternative 


selection and decision structuring as well as the quantification process 
itself. 


Pitfalls and Limitations. With sufficient time to make a decision, 
decision makers often postpone the decision until it is no longer time 
relaxed. There is also a tendency to become "sloppy" if there is too much 
time. Moreover, if there is too much lead time, the decision situation 
_ may change substantially by the time the actual decision is to be made, 
thereby, negating all the decision making efforts. There is sometimes a 
temptation to do something, anything, rather than wait to take action. 


Interactions with Other Decision Elements. No major interactions. 


Prerequisites. None. 
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TIME PRESSURE DECISION MAKING 


General Definition. A decision situation where the decision maker 
has insufficient time to completely structure the decision in terms of 
number of attributes, alternatives, states of nature, and outcome as well 
as alternative selection. In summary there are time constraints and the 
greater the constraints, the greater the pressure. | 


Amplification. In those situations where fast moving events 
preclude adequate consideration of structuring a decision and selecting | 
an alternative, there is time pressured decision making. For example, 

- when an engine quits in the LAMPS helicopter due to an engine fire, there 
is very little time to reflect on the alternative courses of action, 

_ their potential outcomes three or more steps ahead, to predict the various 
relevant states of nature, and then to select a decision rule and use it 
to select an optimal alternative course of action. 


Rules: There are only rules of thumb under time pressure. The 
most obvious one is "Don't take longer than the time available to structure | 
the decision and select an action." If this happens, the decision is out 
of the decision maker's control. The general rule is to spend time on 
structuring the decision at the expense of a hasty action selection. In 
most cases, the wrong decision following a correct structuring is easier 
to recover from than the correct decision on the wrong problem. Also, try 
to focus on the major decision elements immediately, foregoing a close 
look at the secondary considerations. 


Pitfalls and Limitations: The idea of the optimal decision under 
time pressure is illusory. The best that one can hope for in many instances 
is to avoid making a "big" mistake by utilizing the knowledge provided in 
previous trainings. Decision analysis is thought to be worse than making - 
some (or any) decision. 
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Interaction with Other Decision Elements: Time pressure normally 
effects all other decision elements adversely. 


Prerequisites: Training in decision making under pressure can help 
enormously. 
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STATIC DECISION MAKING 


General Definition. A decision situation is static if the. 
consequences of the decision specified through the outcome set are constant 
over time (i.e., not a function of time). 


Amplification. Consider the time horizon over which the 
consequences of a decision unfold. If the consequences (outcome set) 
remain constant over that time horizon, the decision is static. This 
issue is critical sincé the outcome set is estimated in advance of the 
decision and correct representation is required for properly structuring 
the decision. For example, in deciding about the number of sonobuoys to 
drop in a particular pattern, we consider that each number of sonobuoys 
will generate a specific “listening” coverage on the ocean surface. This 
listening coverage remains constant over the time horizon of the decision 
which, in this example, is the lifetime of the particular sonobuoys used. 


Rules: Determine the pattern of decision consequences (outcome) 
over time and if the response pattern is constant, the situation is 
static and a point estimate may be used to represent the outcome. In. 
essence, a "Snapshot" of the consequence may be taken at any point in 
time to represent the outcome. 


Pitfalls and Limitations. In some instances, there will be 
outcomes that are both static and dynamic (vary with time) and they 
must be identified and treated separately. Confusion can be minimized 
if the consequence magnitude is plotted over time to determine if 
magnitude is a function of time. 


Interactions with Other Decision Elements. There is some 


interaction with outcome estimation that follows from attribute (value 
dimension) specification. In some instances, dynamic patterns can be 
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represented by static attributes by careful wording in identifying the 
static attribute. For example, peak response takes a characteristic of 
a dynamic response and represents it by a static attribute. 


Prerequisites: An understanding of how behavior varies as a function 
of time. 
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DYNAMIC DECISION MAKING 


General Definition: A decision situation where the action taken 
produces consequences that vary as a function of time. 


Amplification: Consider the time horizon over which the consequences 
of a decision unfold. If the consequences (outcome set) are not constant 
(i.e., display of pattern over the time horizon) the decision is dynamic. 
This has critical implications for outcome specification since the pattern 
cannot be captured by a single number. The outcome, in fact, may have a 
number of transient states and an eventual equilibrium, each state ees 
the decision to be made in different ways. For example, the ratio of the 
number of active sonobuoys to passive sonobuoys may vary over the time 
horizon of a mission. This is particularly true if the initial ratio of 
buoys (active to passive) is "ideal" for one particular mission. scenario 
in the initial phases, but becomes worse if the scenario unfolds later 
in a manner other than anticipated. . The operator then is faced with a 
new "situation" and may have to reorient his decision attitude. In 
the event of engine failure, the "dangerousness" of the decision outcome 
will change as the craft passes from over water to over land. If the 
initial failure occurs over water and the pilot decides to try and make 
jit home on one engine, the decision outcome is at one “level of danger" 
over water and another, lower "level of danger" over land. Care must be 
given to using the respective durations in trying to develop numbers to | 
represent the decision outcomes, not just a single point estimate. 


Rules: Determine the pattern of decision consequences (outcomes) 
over time and if the response pattern changes over:time, the situation 
is dynamic and the entire response pattern and its characteristics must 
be considered in order to represent the outcome. Characteristics of 
response patterns such as peak and valley magnitudes, durations of peaks 
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MAXI-MIN UTILITY 


General Description: A payoff table is constructed (with any 
multiattribute outcomes aggregated to a single utility) and expressed 
in utilities, then the alternative is chosen that maximizes the minimum 
utility. 


Amplification: Each alternative is arrayed for each state of 
nature. The outcomes are all expressed in utilities. Since nothing 
is known or assumed (or estimated) about the probabilities of the various 
states of nature, each alternative is examined to find the worst outcome. 
When the worst case utility is identified for each alternative, the 
alternative with maximum "worst" utility is chosen. Therefore, no 
matter what state of nature eventually occurs, the utility of the 
outcomes can be no less than the minimum utility of the alternative 
chosen. The decision rule guarantees the best of the worst, or maximizes 
the minimum utility. Hence, a pessimistic decision maker is always 
covered against the worst case. 


Rules: Construct the payoff matrix and estimate the utilities 
of each alternative under each state of nature. Then identify the lowest 
utility for each alternative under any state of nature for each 
alternative and select the alternative with the highest utility. 


Pitfalls and Limitations: The method assumes that nothing is 
known about the relative probabilities of the states of nature and, 
hence, assumes that each are equally likely. This may not be so in 
a given decision situation. Also, the method does not lead to an 
optimal decision in the expected utility sense. 


Prerequisites: Training in utility estimation and aggregation 
and payoff table construction are prerequisites. 
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MAXT-MAX UTILITY 


General Description: A payoff table is constructed with all 
outcomes (single and multiattribute) represented by a single utility 
and then the alternative that maximizes the maximum utility is chosen. 


Amplification: Each alternative is arrayed for each state of 
nature and the outcomes are all expressed in utilities. Since nothing 
is known, assumed, or estimated about the probabilities of the various. 
states of nature, each alternative is examined to find the best outcome 
_(i.e., the maximum utility). Then the highest utility is identified 
for each alternative (regardless of the state of nature under which 
it occurs) the alternative with the maximum, highest utility is chosen. 
This decision rule optimistically assumes that if the best happens, the 
alternative will have been chosen to deliver the maximum utility. 


Rules: Construct the payoff matrix and estimate the utilities 
of each alternative under each state of nature. Then identify the 
highest utility for each alternative regardless of the state of nature 
under which it occurs, and select the alternative with the maximum 
(highest) utility. 


Pitfalls and Limitations: This method again assumes that nothing 
is known about the probabilities associated with the various states of 
nature, and this may not be so. Also, this method may reward 
optimists, but there is no guarantee against losses or even that the 
optimal alternative is chosen in a SEU sense. 

es | 

Interaction with Other Decision Elements: Under time relaxed, 

no interactions. Under time pressure, this method has similar 


advantages computationally to those of maxi-min utility. 
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Prerequisites: Training is utility assessment and aggregation and 
in payoff table construction. 
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MINI-MAX REGRET 


General Description: A payoff table is constructed and from it, 
a regret matrix is constructed showing how much "regret" the decision 
maker would have if he made the wrong decision. The alternative is 
chosen that minimizes the maximum regret of the decision maker. 


Amplification: Each alternative is arrayed in a payoff table as 
in maxi-min utility. A regret matrix is constructed by finding the best 
possible alternative action under any state of nature. If that state of 
nature turned up, the decision maker would have no regret. If he had . 
chosen any other alternative under that state of nature, he would regret 
it by the amount of the difference between the best he could do and the 
amount for the chosen alternative. A regret matrix is constructed 
comparing how much a decision maker would regret his decision if the best 
occurred under each state of nature (i.e., he had chosen the best alternative 
for state of nature). The regrets for each alternative under each state 
of nature form the regret matrix. Each alternative is scanned to find 
the maximum regret. The alternative with the smallest maximum regret 
is selected. 


Rules: Construct the payoff table. Then, construct the regret 
matrix, identify the maximum regrets, and select the alternative with 
smallest maximum regret. . 


Pitfalls and Limitations: In this method, the addition of a new 
alternative that itself is not eventually chosen, can shift the decision 
to a different alternative other than the one chosen before the addition. 
Also, the method assumes that nothing is known about probabilities of 
states of nature. 
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Interaction with Other Decision Elements: Under time relaxed, no 


St a SSS 


significant interactions. Under time pressure the computations are time 


consuming. 


Prerequisites: Training in utility assessment and aggregation and 
in payoff and regret matrix construction. 
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LEXICOGRAPHY 


General Description: Lexicography, as used here, is a procedure . 
whereby the most important attribute of a decision situation is considered 
for each alternative and the decision is made on the basis of which 
alternative scores the highest in this attribute. In the event of a tie 
on the most important, the next most important attribute is identified 
and the alternative that scores the highest on it is selected. In the 
event of a tie the third most important attribute is identified and the 
process repeats until an alternative is chosen. Only those alternatives 
that tie are considered in subsequent rounds. 


Amplification: Lexicography is like looking up a word in the 
dictionary. You start with the first letter, then the second and so on 
until you have found the word you are looking for. Similarly in 
lexicographical decision making you identify the most important attribute, 
and then make a decision on it unless there is a tie. In the event of a 
tie the process repeats on the second most important then the third, 
and so on. The process works when there are multiattribute alternatives. 
For example, in the event of an engine fire in a LAMPS helicopter, the 
pilot must consider safety, mission completion, and so on in deciding 
whether to ditch, or attempt to restart. If safety is the most important 
factor, the decision would be made by selecting the alternative 
(ditching or restart in this example) that scores the highest on this 
attribute. If the decision cannot be made on the basis of safety alone, 
mission completion would then be considered, etc. 


Rules: Identify the relative importance of each attribute. 
Follow the lexicographical procedures. 
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Pitfalls and Limitations: A serious limitation in this technique 
is that the second and third most important attributes together may 
outweigh the first most important one. Hence, clearly suboptimal 
alternatives may be selected. This method assumes that only one dimension 
at a time is relevant for decision making, and that it is more important 
than all the others. 


Interaction with Other Decision Elements: Under time relaxed, 


no significant interactions. Under time pressure the computations are 
so easy that there are no significant interactions. 


Prerequisites: Training is establishing the relative importance 
of the attribute dimensions in multiattributed decision alternatives. 
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HURWICZ 


General Description: A payoff table is constructed so that each 
alternative is represented by a single utility number of each state of 
nature. Then, for each alternative the best and worse (highest and lowest) 
utilities are identified. This represents the best and worst case (most 
optimistic and most pessimistic). The alternative that has the best 
"combination" of best and worst utilities is selected. 


Amplification: The attempt here is to combine a pessimistic and 
optimistic approach. The decision maker establishes how likely the best 
and worst cases are to occur (by establishing probabilities that sum to 
one for both). The expected utility is then calculated using only the 
best and worst utilities and their respective probability estimates. 


Rules: Establish the payoff table and identify the best and worst 
utility for each outcome. Estimate the probabilities for each (sum to 
one). Then calculate the expected utility for each alternative. 


Pitfalls and Limitations: This approach examines only the best 
and worst. The other outcomes are ignored. Also, the estimate of the 
probabilities of each of the best and worst occurring is subject to error. 


Interaction with Other Decision Elements: Under time relaxed, no 
interactions. Under time pressure, there may not be time to properly — 
measure the probabilities. 


Prerequisites: Constructing payoff table. Assigning and 


aggregating utilities. Estimating coefficients of optimism:and 
pessimism (the probabilities). 
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SATISFYING 


General Description: Establish the feasible domain for all 
alternatives. Select the first alternative screened that falls into 
the feasible domain. 


Amplification: For multi-attribute alternatives, establish the 
minimum (or maximum) acceptable boundaries of performance for each 
attribute. This defines the feasible domain. Any alternative screened 
that falls outside of these limits on any attribute dimensions is 
eliminated from further consideration. The first alternative that 
meets all the limits stated is selected without further consideration. 
of any additional alternatives. 


. Rules: Establish the minimum and/or maximum thresholds for each 

attribute value dimension. Plot each alternative considered on the 
feasible domain and select the first to fall completely within the 
feasible domain. 


Pitfalls and Limitations: The method produces an alternative, but 
it may be the worst of all possible alternative that are feasible, and 
hence far from optimal. 


Interaction with Other Decision Elements: Under time relaxed, no 
interactions. Under time pressure, if the feasible domain boundaries 
can be established in time, the method will produce a rapid decision 
on an alternative that is feasible and meets al] minimums (maximums) 
required. . 


Prerequisites: Training at establishing standard minimums and 
maximums for alternative dimensions. 
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